Storage medium having stored thereon information processing program, information processing apparatus, information processing system, and information processing method

ABSTRACT

On the basis of data output from a portable display apparatus, a direction of rotation of the portable display apparatus about a predetermined direction in real space is calculated. Attitudes of an object, placed in a virtual world, and a first virtual camera, for generating an image of the virtual world including at least the object, are controlled such that the object and the first virtual camera rotate in the direction of rotation about a direction that corresponds to the predetermined direction and is set in the virtual world. Then, a first image representing the virtual world viewed from the first virtual camera is displayed on the portable display apparatus.

CROSS REFERENCE TO RELATED APPLICATION

The disclosures of Japanese Patent Application No. 2011-050039, filed onMar. 8, 2011, Japanese Patent Application No. 2011-083453, JapanesePatent Application No. 2011-083454, Japanese Patent Application No.2011-083455, and Japanese Patent Application No. 2011-083456, filed onApr. 5, 2011, Japanese Patent Application No. 2011-115402, JapanesePatent Application No. 2011-115403, and Japanese Patent Application No.2011-115404, filed on May 24, 2011, Japanese Patent Application No.2011-118901 and Japanese Patent Application No. 2011-118902, filed onMay 27, 2011, Japanese Patent Application No. 2011-123644, JapanesePatent Application No. 2011-123645, and Japanese Patent Application No.2011-123646, filed on Jun. 1, 2011, and Japanese Patent Application No.2011-225538, filed on Oct. 13, 2011, are incorporated herein byreference.

FIELD

The technique shown here relates to a storage medium having storedthereon an information processing program, an information processingapparatus, an information processing system, and an informationprocessing method, and in particular, relates to a storage medium havingstored thereon an information processing program that, for example,performs the process of displaying a virtual world, and an informationprocessing apparatus, an information processing system, and aninformation processing method that, for example, perform the process ofdisplaying a virtual world.

BACKGROUND AND SUMMARY

Conventionally, there is a game where a user operates a hand-held gameapparatus while holding it, and causes the hand-held game apparatus todisplay a game screen based on the operation. The hand-held gameapparatus generates a game image on the basis of various operationsperformed on an operation key, a touch panel, and the like by the user,and advances a game by displaying a game image on a display sectionprovided in the hand-held game apparatus.

In the hand-held game apparatus, however, the game image is displayedwith respect to directions defined in advance in the display section ofthe hand-held game apparatus. Thus, if the user changes the direction inwhich they are holding the hand-held game apparatus, it becomesdifficult to view the game image displayed on the display section.

Therefore, it is an object of the exemplary embodiment to provide astorage medium having stored thereon an information processing programcapable of, when an image of a virtual world is displayed on a displayapparatus that allows a user to view a screen thereof while holding it,displaying a suitable image on the apparatus, regardless of thedirection in which the user is holding the apparatus, and an informationprocessing apparatus, an information processing system, and aninformation processing method that are capable of, when an image of avirtual world is displayed on a display apparatus that allows a user toview a screen thereof while holding it, displaying a suitable image onthe apparatus, regardless of the direction in which the user is holdingthe apparatus.

To achieve the above object, the exemplary embodiment may employ, forexample, the following configurations. It is understood that when thedescription of the scope of the appended claims is interpreted, thescope should be interpreted only by the description of the scope of theappended claims. If the description of the scope of the appended claimscontradicts the description of these columns, the description of thescope of the appended claims has priority.

In an exemplary configuration of a computer-readable storage mediumhaving stored thereon an information processing program according to theexemplary embodiment, the information processing program is executed bya computer of an information processing apparatus capable of displayingan image on a portable display apparatus that outputs at least databased on an attitude and/or a motion of the portable display apparatusbody. The information processing program causes the computer to execute:calculating, on the basis of the data output from the portable displayapparatus, a direction of rotation of the portable display apparatusabout a predetermined direction in real space; controlling an attitudeof an object, placed in a virtual world, such that the object rotates inthe direction of rotation about a direction that corresponds to thepredetermined direction and is set in the virtual world; controlling anattitude of a first virtual camera, for generating an image of thevirtual world including at least the object, such that the first virtualcamera rotates in the direction of rotation about the direction thatcorresponds to the predetermined direction and is set in the virtualworld; and displaying on the portable display apparatus a first imagerepresenting the virtual world viewed from the first virtual camera.

It should be noted that the information processing apparatus may be anapparatus that performs game processing and generates an image based onthe game processing, or may be a versatile apparatus such as a generalpersonal computer. The portable display apparatus may have a size smallenough to be carried by a user. Typically, the portable displayapparatus may be a display apparatus that allows the user to view animage displayed thereon by holding it with both hands. Further, as in aterminal apparatus according to the embodiment described later, theportable display apparatus may or may not include components other than:means for outputting at least data based on the attitude and/or themotion of the portable display apparatus body; and means for displayingthe first image.

Based on the above, when an image of an virtual world is displayed on aportable display apparatus, it is possible to display a suitable imageon the portable display apparatus, regardless of the direction in whicha user is holding the portable display apparatus. Further, it ispossible to display a suitable image in accordance with the attitude ofthe portable display apparatus, and simultaneously control the attitudeof an object in a virtual world displayed on the portable displayapparatus.

In addition, an amount of rotation of the portable display apparatusabout the predetermined direction in real space may be furthercalculated on the basis of the data output from the portable displayapparatus. The attitude of the object may be controlled such that theobject rotates in the direction of rotation about the direction thatcorresponds to the predetermined direction and is set in the virtualworld, and by an amount of rotation based on the amount of rotation ofthe portable display apparatus. The attitude of the first virtual cameramay be controlled such that the first virtual camera rotates in thedirection of rotation about the direction that corresponds to thepredetermined direction and is set in the virtual world, and by anamount of rotation based on the amount of rotation of the portabledisplay apparatus.

Based on the above, it is possible to display a suitable image inaccordance with the amount of rotation of the portable displayapparatus, and simultaneously control the attitude of the object in thevirtual world displayed on the portable display apparatus.

In addition, the attitude of the first virtual camera may be controlledsuch that the first virtual camera rotates in the direction of rotationby the same amount of rotation as the amount of rotation of the portabledisplay apparatus.

Based on the above, a virtual camera for generating the virtual world tobe displayed on the portable display apparatus is caused to rotate inthe same direction of rotation and by the same amount of rotation as thedirection of rotation and the amount of rotation in and by which theportable display apparatus rotates. This enables the user to enjoy afeeling as if peeping at the virtual world through the portable displayapparatus.

In addition, the information processing program may further cause thecomputer to execute calculating, on the basis of the data output fromthe portable display apparatus, a direction of gravity relative to theportable display apparatus. The direction of rotation and the amount ofrotation of the portable display apparatus may be calculated withrespect to the direction of gravity. The attitude of the first virtualcamera may be controlled such that when the first image is displayed onthe portable display apparatus, the direction of gravity relative to theportable display apparatus displaying the first image coincides with adirection of gravity in the virtual world displayed as the first image.

Based on the above, it is possible to change the direction of thevirtual world displayed on the portable display apparatus so as tocoincide with the direction of gravity applied to the portable displayapparatus. This enables the user to enjoy a feeling as if real spacewere the virtual world.

In addition, the object may be controlled so as to rotate in thedirection of rotation and by the same amount of rotation as the amountof rotation of the portable display apparatus, and to have such anattitude and position that a positional relationship between the objectand the first virtual camera is fixed.

Based on the above, it is possible to display the object of the virtualworld on a display screen of the portable display apparatus at apredetermined position. This facilitates the control of the object basedon the attitude of the portable display apparatus.

In addition, using as the predetermined direction a predetermined axisprovided in the portable display apparatus, the direction of rotationand the amount of rotation of the portable display apparatus about thepredetermined axis may be calculated.

Based on the above, the portable display apparatus is moved and theattitude of the portable display apparatus is changed, so as to rotateat least about a predetermined axis provided in the portable displayapparatus, whereby it is possible to control the attitudes of the firstvirtual camera and the object.

In addition, using as the predetermined axis a perspective direction ofa display screen that displays the first image on the portable displayapparatus, the direction of rotation and the amount of rotation ofportable display apparatus about the perspective direction may becalculated. The attitude of the first virtual camera may be controlledsuch that, using a direction of a line of sight of the first virtualcamera as a direction corresponding to the perspective direction, thefirst virtual camera rotates about the direction of the line of sight,in the direction of rotation and by the same amount of rotation as theamount of rotation of the portable display apparatus.

Based on the above, the portable display apparatus is moved and theattitude of the portable display apparatus is changed, so as to roll atleast about the perspective direction of the display screen of theportable display apparatus, whereby it is possible to roll the attitudesof the first virtual camera and the object about the direction of theline of sight of the first virtual camera.

In addition, further using, as the predetermined axis, each of a heightdirection and a width direction of the portable display apparatus thatare orthogonal to the perspective direction and are orthogonal to eachother, the direction of rotation and the amount of rotation of theportable display apparatus about each of the height direction and thewidth direction may be calculated. The attitude of the first virtualcamera may be controlled such that: the first virtual camera rotatesabout a height direction orthogonal to the direction of the line ofsight of the first virtual camera, in the same direction of rotation andby the same amount of rotation as the direction of rotation and theamount of rotation about the height direction of the portable displayapparatus; and the first virtual camera rotates about a width directionorthogonal to the direction of the line of sight of the first virtualcamera and orthogonal to the height direction orthogonal to thedirection of the line of sight of the first virtual camera, in the samedirection of rotation and by the same amount of rotation as thedirection of rotation and the amount of rotation of the portable displayapparatus about the width direction of the portable display apparatus.

Based on the above, in addition to roll about the perspective directionof the display screen, the portable display apparatus is moved and theattitude of the portable display apparatus is changed so as to pitch andyaw, whereby it is also possible to control the attitudes of the firstvirtual camera and the object.

In addition, the attitude of the first virtual camera in the virtualworld may be controlled so as to be the same as the attitude of theportable display apparatus in real space.

Based on the above, the attitude of the first virtual camera iscontrolled so as to be the same as the attitude of the portable displayapparatus, whereby, in accordance with the user directing the portabledisplay apparatus in the direction that they wish to view, it ispossible to provide the user with, for example, an image as if peepingat the virtual world through the portable display apparatus. This makesit possible to provide the user with a feeling as if being in thevirtual world.

In addition, image data indicating the first image may be output to theportable display apparatus. The portable display apparatus may includean image data acquisition unit and a display unit. The image dataacquisition unit acquires the image data output from the informationprocessing apparatus. The display unit displays the first imageindicated by the image data acquired by the image data acquisition unit.

Based on the above, the portable display apparatus can function as aso-called thin-client terminal, which does not perform informationprocessing.

In addition, the information processing program may further cause thecomputer to execute generating compression image data by compressing theimage data indicating the first image. In this case, the generatedcompression image data may be output to the portable display apparatus.The compression image data output from the information processingapparatus may be acquired. The portable display apparatus may furtherinclude a display image decompression unit. The display imagedecompression unit decompresses the compression image data to obtain theimage data indicating the first image. The display unit may display thefirst image indicated by the image data that has been acquired by theimage data acquisition unit and has been decompressed by the displayimage decompression unit.

Based on the above, the first image is decompressed before being outputfrom the information processing apparatus to the portable displayapparatus. This makes it possible to output the first image at a highspeed, and reduce delay caused between the generation of the first imageand the display of the first image on the portable display apparatus.

In addition, besides the first image, a second image representing thevirtual world viewed from a second virtual camera may be furtherdisplayed on another display apparatus connected to the informationprocessing apparatus.

It should be noted that said another display apparatus described aboveis a display apparatus connected to the information processingapparatus, like a monitor 2 according to the embodiment described later.Said another display apparatus may be a component separate from theportable display apparatus, and may be any apparatus so long as it iscapable of displaying the second image generated by the informationprocessing apparatus. For example, said another display apparatusdescribed above may be integrated with the game apparatus (in a singlehousing).

Based on the above, when processing based on the operation of moving andchanging the attitude of the portable display apparatus is performed, itis possible to display the results of the processing not only on theportable display apparatus but also on said another display apparatusconnected to the information processing apparatus. This enables the userto use, in accordance with the state of the operation or the user'spreference, either one of images displayed on, for example, twoapparatuses, and also view an image suitable for an operation of theuser. Further, it is possible to use an image displayed on said anotherdisplay apparatus connected to the information processing apparatus, as,for example, an image to be viewed by another person different from theuser. This makes it possible to provide a viewing environment suitablealso for the case where a plurality of people view the results of theprocessing.

In addition, the information processing program may further cause thecomputer to execute generating compression image data by compressing theimage data indicating the first image. In this case, the generatedcompression image data may be output to the portable display apparatus,and, besides the compression image data, image data indicating thesecond image may be output to said another display apparatus withoutbeing compressed. The portable display apparatus may include an imagedata acquisition unit, a display image decompression unit, and a displayunit. The image data acquisition unit acquires the compression imagedata output from the information processing apparatus. The display imagedecompression unit decompresses the compression image data to obtain theimage data indicating the first image. The display unit displays thefirst image indicated by the image data that has been acquired by theimage data acquisition unit and has been decompressed by the displayimage decompression unit.

Based on the above, the first image is decompressed and then output fromthe information processing apparatus to the portable display apparatus.This makes it possible to output the first image at a high speed, andreduce delay caused between the generation of the first image and thedisplay of the first image on the portable display apparatus.

In addition, the information processing program may further cause thecomputer to execute setting, on the basis of a position of the object inthe virtual world, the second virtual camera, for generating an image ofthe virtual world, at a position different from a position of the firstvirtual camera such that the object is included in the second image.

Based on the above, the same object is displayed not only on theportable display apparatus but also on said another display apparatus,and images of the virtual world that are different in the point of vieware displayed thereon. This enables the user to use, in accordance withthe state of the operation or the user's preference, either one of theimages displayed on the two apparatuses when controlling the object.

In addition, the second virtual camera may be set at a position furtheraway from the object than the first virtual camera is from the object. Arange wider than a range of the virtual world represented by the firstimage may be displayed as the second image on said another displayapparatus.

Based on the above, an image of the virtual world in a display rangewider than that of an image of the virtual world displayed on theportable display apparatus is displayed on said another displayapparatus connected to the information processing apparatus. This makesit possible to display on each display apparatus, for example, an imagesuitable for an operation of the user when the state of the virtualworld is presented to the user.

In addition, the second virtual camera may be set at a position ofviewing the object from a bird's-eye view in the virtual world. An imageobtained by viewing from a bird's-eye view the object placed in thevirtual world may be displayed as the second image on said anotherdisplay apparatus.

Based on the above, an image of the virtual world based on the attitudeof the portable display apparatus is displayed on the portable displayapparatus, and an image of the virtual world obtained by looking downupon it is displayed on another display apparatus connected to theinformation processing apparatus. This makes it possible to display oneach display apparatus, for example, an image suitable for an operationof the user when the state of the virtual world is presented to theuser.

In addition, the portable display apparatus may include at least one ofa gyro sensor and an acceleration sensor, each of which outputs databased on the attitude and/or the motion of the portable displayapparatus body. In this case, on the basis of the data output from theat least one of the gyro sensor and the acceleration sensor, thedirection of rotation and an amount of rotation of the portable displayapparatus may be calculated.

Based on the above, using the data that is output from the gyro sensorand indicates the angular velocity generated in the portable displayapparatus and/or the data that is output from the acceleration sensorand indicates the acceleration generated in the portable displayapparatus, it is possible to accurately calculate the attitude and themotion of the portable display apparatus.

In addition, the exemplary embodiment may be carried out in the forms ofan information processing apparatus and an information processingsystem, each including units that perform the above processes, and aninformation processing method including the above operations.

The exemplary embodiment makes it possible to, when an image of avirtual world is displayed on a portable display apparatus, display asuitable image on the portable display apparatus, regardless of thedirection in which the user is holding the portable display apparatus.

These and other objects, features, aspects and advantages of theexemplary embodiment will become more apparent from the followingdetailed description when taken in conjunction with the accompanyingdrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an external view showing a non-limiting example of a gamesystem 1;

FIG. 2 is a functional block diagram showing a non-limiting example of agame apparatus body 5 of FIG. 1;

FIG. 3 is a diagram showing a non-limiting example of the externalconfiguration of a terminal apparatus 6 of FIG. 1;

FIG. 4 is a diagram showing a non-limiting example of the state where auser holds the terminal apparatus 6;

FIG. 5 is a block diagram showing a non-limiting example of the internalconfiguration of the terminal apparatus 6 of FIG. 3;

FIG. 6 is a perspective view showing a non-limiting example of theappearance of a board-type controller 9 of FIG. 1;

FIG. 7 is a diagram showing a non-limiting example of a cross-sectionalview of the board-type controller 9 shown in FIG. 6 taken along lineA-A, and a non-limiting example of an enlarged view of a corner portionwhere a load sensor 94 is arranged;

FIG. 8 is a block diagram showing a non-limiting example of theelectrical configuration of the board-type controller 9 of FIG. 6;

FIG. 9 is a diagram showing a non-limiting example of the state of auser performing an operation using the terminal apparatus 6 and theboard-type controller 9;

FIG. 10A is a diagram showing a non-limiting example of an imagedisplayed on a LCD 61 of the terminal apparatus 6;

FIG. 10B is a diagram showing a non-limiting example of an imagedisplayed on a monitor 2;

FIG. 11A is a diagram showing a non-limiting example of an imagedisplayed on the LCD 61 in the case where the terminal apparatus 6 isviewed in a squarely-viewed state;

FIG. 11B is a diagram showing a non-limiting example where the imageshown in FIG. 11A is displayed on the LCD 61 of the terminal apparatus 6in the squarely-viewed state;

FIG. 12A is a diagram showing a non-limiting example of an imagedisplayed on the LCD 61 in the case where the terminal apparatus 6 hasbeen roll-rotated counterclockwise from the squarely-viewed state;

FIG. 12B is a diagram showing a non-limiting example where the imageshown in FIG. 12A is displayed on the LCD 61 of the terminal apparatus 6roll-rotated counterclockwise from the squarely-viewed state;

FIG. 13A is a diagram showing a non-limiting example of an imagedisplayed on the LCD 61 in the case where the terminal apparatus 6 hasbeen roll-rotated counterclockwise from the squarely-viewed state and isviewed in a vertically-held state;

FIG. 13B is a diagram showing a non-limiting example where the imageshown in FIG. 13A is displayed on the LCD 61 of the terminal apparatus 6in the vertically-held state;

FIG. 14 is a diagram illustrating non-limiting examples of: therelationship between a terminal apparatus perspective directionprojected onto a vertical plane in real space and an operationindication direction projected onto a vertical plane in a virtual world;and a rotational movement corresponding to a terminal apparatus updirection (a roll angle);

FIG. 15 is a diagram showing a non-limiting example of data and programsthat are stored in a main memory of the game apparatus body 5 of FIG. 1;

FIG. 16 is a flow chart showing a non-limiting example of informationprocessing performed by the game apparatus body 5 of FIG. 1;

FIG. 17 is a subroutine flow chart showing a non-limiting example of agame control process in step 44 in FIG. 16;

FIG. 18 is a subroutine flow chart showing a non-limiting example of aplayer object setting process in step 83 in FIG. 17;

FIG. 19 is a subroutine flow chart showing a non-limiting example of aplayer object action setting process in step 127 in FIG. 18; and

FIG. 20 is a diagram illustrating a non-limiting example of therelationships among: a real space reference direction and the terminalapparatus perspective direction that are projected onto a horizontalplane in real space; and a virtual world reference direction and theoperation indication direction that are projected onto a horizontalplane in the virtual world.

DETAILED DESCRIPTION OF NON-LIMITING EXAMPLE EMBODIMENTS

With reference to FIG. 1, an information processing apparatus forexecuting an information processing program according to an exemplaryembodiment and an information processing system including theinformation processing apparatus is described. Hereinafter, in order toprovide a specific description, a stationary game apparatus body 5 isused as an example of the information processing apparatus, and a gamesystem including the game apparatus body 5 is described. FIG. 1 is anexternal view showing an example of the game system 1 including thestationary game apparatus body 5. FIG. 2 is a block diagram showing anexample of the game apparatus body 5. Hereinafter, the game system 1 isdescribed.

As shown in FIG. 1, the game system 1 includes a household televisionreceiver (hereinafter referred to as a “monitor”) 2 which is an exampleof display means, and the stationary game apparatus 3 connected to themonitor 2 via a connection cord. The monitor 2 includes loudspeakers 2 afor outputting, in the form of sound, a sound signal outputted from thegame apparatus 3. Further, the game apparatus 3 includes: an opticaldisk 4 having stored therein a program (e.g., a game program), which isan example of the information processing program according to theexemplary embodiment; the game apparatus body 5 having a computer forexecuting the program stored in the optical disk 4 to display a gamescreen on the monitor 2; a terminal apparatus 6; a controller 7 forproviding the game apparatus body 5 with operation information used tooperate, for example, objects displayed on the display screen; and aboard-type controller 9. The game system 1 performs game processing onthe game apparatus body 5 in accordance with a game operation using atleast one of the terminal apparatus 6, the controller 7, and theboard-type controller 9, and displays a game image obtained by the gameprocessing on the monitor 2 and/or the terminal apparatus 6. The gameapparatus body 5 is wirelessly connected to the terminal apparatus 6,the controller 7, and the board-type controller 9 so as to enablewireless communication therebetween. For example, the wirelesscommunication is performed according to the Bluetooth (registeredtrademark) standard or the IEEE 802.11n standard. The wirelesscommunication, however, may be performed in accordance with otherstandards such as standards for infrared communication.

The optical disk 4, typifying an information storage medium used for thegame apparatus body 5 in an exchangeable manner, is detachably insertedin the game apparatus body 5. The optical disk 4 has stored therein theinformation processing program (typically, a game program) to beperformed by the game apparatus body 5. The game apparatus body 5 has,on a front surface thereof, an insertion opening for the optical disk 4.The game apparatus body 5 reads and executes the information processingprogram stored in the optical disk 4 inserted into the insertion openingto perform the information processing.

The monitor 2 is connected to the game apparatus body 5 via a connectioncord. The monitor 2 displays a game image obtained by the gameprocessing performed by the game apparatus body 5. The monitor 2includes the loudspeakers 2 a. The loudspeakers 2 a each output a gamesound obtained as a result of the game processing. In anotherembodiment, the game apparatus body 5 and a stationary display apparatusmay be integrated with each other. The communication between the gameapparatus body 5 and the monitor 2 may be wireless communication.

The game apparatus body 5 has mounted thereto a flash memory 17 (seeFIG. 2) which functions as a backup memory for fixedly storing data suchas saved data. The game apparatus body 5 executes the game program orthe like stored in the optical disk 4, and displays a result thereof asa game image on the monitor 2 and/or the terminal apparatus 6. The gameprogram or the like to be executed may be stored in advance in the flashmemory 17 as well as in the optical disk 4. Further, the game apparatusbody 5 may reproduce a state of a game played in the past, using thesaved data stored in the flash memory 17, and display an image of thegame state on the monitor 2 and/or the terminal apparatus 6. A user ofthe game apparatus 3 can enjoy the game progress by operating at leastone of the terminal apparatus 6, the controller 7, and the board-typecontroller 9 while viewing the game image displayed on the monitor 2and/or the terminal apparatus 6.

The controller 7 and the board-type controller 9 each wirelesslytransmit transmission data such as operation information, using, forexample, the Bluetooth technology, to the game apparatus body 5 having acontroller communication module 19. The controller 7 is operation meansfor performing, for example, selection of options displayed on thedisplay screen of the monitor 2. The controller 7 includes a housingwhich is small enough to be held by one hand, and a plurality ofoperation buttons (including a cross key and the like) which are exposedat the surface of the housing. In addition, as is described later, thecontroller 7 includes an imaging information calculation section fortaking an image viewed from the controller 7. As exemplary imagingtargets of the imaging information calculation section, two LED modules(hereinafter referred to as “markers”) 8L and 8R are provided in thevicinity of the display screen of the monitor 2 (above the screen inFIG. 1). Although details will be described later, a user (player) isallowed to perform a game operation while moving the controller 7, andthe game apparatus body 5 uses a marker 8 to calculate the movement,position, attitude and the like of the controller 7. The marker 8 hastwo markers 8L and 8R at both ends thereof. Specifically, the marker 8L(as well as the marker 8R) includes one or more infrared LEDs (LightEmitting Diodes), and emits infrared light forward from the monitor 2.The marker 8 is connected to the game apparatus body 5, so that the gameapparatus body 5 can control the infrared LEDs included in the marker 8to be lit on or off. The marker 8 is a portable unit, so that the useris allowed to place the marker 8 in a given position. Although FIG. 1shows a case where the marker 8 is placed on the monitor 2, the locationand direction of the marker 8 may be appropriately selected. Further,the controller 7 is capable of receiving, at a communication section,transmission data wirelessly transmitted from the controllercommunication module 19 of the game apparatus body 5, to generate asound or vibration based on the transmission data.

In another embodiment, the controller 7 and/or the board-type controller9 may be wire-connected to the game apparatus body 5. Further, in theexemplary embodiment, the game system 1 includes a controller 7 and aboard-type controller 9. The game apparatus body 5, however, is capableof communicating with a plurality of controllers 7 and a plurality ofboard-type controllers 9. Therefore, a plurality of players can play agame using a predetermined number of controllers 7 and board-typecontroller 9 simultaneously.

The controller 7 includes a housing which is formed by, for example,plastic molding, and has a plurality of operation sections (operationbuttons) in the housing 71. Then, the controller 7 transmits, to thegame apparatus body 5, operation data indicating the states of inputsprovided to the operation sections (indicating whether or not eachoperation button has been pressed).

In addition, the controller 7 has the imaging information calculationsection that analyzes image data of an image captured by capturing meansand determines an area having a high brightness, and thereby calculatesthe position of the center of gravity, the size, and the like of thearea. For example, the imaging information calculation section hascapturing means fixed in the housing of the controller 7, and uses as animaging target a marker that outputs infrared light, such as a markersection 65 of the terminal apparatus 6 and/or the marker 8. The imaginginformation calculation section calculates the position of the imagingtarget in a captured image captured by the capturing means, andtransmits, to the game apparatus body 5, marker coordinate dataindicating the calculated position. The marker coordinate data variesdepending on the direction (the angle of tilt) or the position of thecontroller 7, and therefore, the game apparatus body 5 can calculate thedirection and the position of the controller 7 using the markercoordinate data.

In addition, the controller 7 includes therein an acceleration sensorand/or a gyro sensor. The acceleration sensor detects the accelerationgenerated in the controller 7 (including the gravitationalacceleration), and transmits, to the game apparatus body 5, dataindicating the detected acceleration. The acceleration detected by theacceleration sensor varies depending on the direction (the angle oftilt) or the movement of the controller 7, and therefore, the gameapparatus body 5 can calculate the direction and the movement of thecontroller 7 using the acquired acceleration data. The gyro sensordetects the angular velocities generated about three axes set in thecontroller 7, and transmits, to the game apparatus body 5, angularvelocity data indicating the detected angular velocities. Theacceleration detected by the gyro sensor varies depending on thedirection (the angle of tilt) or the movement of the controller 7, andtherefore, the game apparatus body 5 can calculate the direction and themovement of the controller 7 using the acquired acceleration data. Asdescribed above, the user is allowed to perform a game operation bypressing any of the operation sections 72 provided on the controller 7,and moving the controller 7 so as to change the position and theattitude (tilt) thereof.

The controller 7 has a loudspeaker and a vibrator. The controller 7processes sound data transmitted from the game apparatus body 5, andoutputs sound corresponding to the sound data from the loudspeaker.Further, the controller 7 processes vibration data transmitted from thegame apparatus body 5, and generates vibration by actuating the vibratorin accordance with the vibration data. It should be noted that in theexemplary embodiment described later, it is possible to play a gamewithout using the controller 7. A detailed configuration of theboard-type controller 9 will be described later.

The terminal apparatus 6 is a portable apparatus that is small enough tobe held by the user, and the user is allowed to move the terminalapparatus 6 with hands, or place the terminal apparatus 6 at anylocation. Although a detailed configuration of the terminal apparatus 6will be described later, the terminal apparatus 6 includes an LCD(Liquid Crystal Display) 61 as display means, and input means (a touchpanel 62, a gyro sensor 604, and the like described later). The terminalapparatus 6 and the game apparatus body 5 (a terminal communicationmodule 28 (see FIG. 2)) are capable of communicating with each otherwirelessly or wired. The terminal apparatus 6 receives, from the gameapparatus body 5, data of an image (e.g., a game image) generated in thegame apparatus body 5, and displays the image represented by the data onan LCD 61. Although the LCD 61 is used as a display apparatus in theexemplary embodiment, the terminal apparatus 6 may include a given otherdisplay apparatus, such as a display apparatus utilizing EL (ElectroLuminescence), for example. Further, the terminal apparatus 6 transmits,to the game apparatus body 5 having the terminal communication module28, operation data representing the content of an operation performed onthe terminal apparatus 6.

Next, with reference to FIG. 2, the internal configuration of the gameapparatus body 5 is described. FIG. 2 is a block diagram showing anexample of the internal configuration of the game apparatus body 5. Thegame apparatus body 5 includes a CPU (Central Processing Unit) 10, asystem LSI (Large Scale Integration) 11, an external main memory 12, aROM/RTC (Read Only Memory/Real Time Clock) 13, a disk drive 14, an AV-IC(Audio Video-Integrated Circuit) 15 and the like.

The CPU 10, serving as a game processor, executes a program stored inthe optical disk 4 to perform a process. The CPU 10 is connected to thesystem LSI 11. In addition to the CPU 10, the external main memory 12,the ROM/RTC 13, the disk drive 14, and the AV-IC 15 are connected to thesystem LSI 11. The system LSI 11 performs processes such as control ofdata transmission between the respective components connected thereto,generation of an image to be displayed, and acquisition of data from anexternal apparatus. The internal configuration of the system LSI 11 willbe described later. The external main memory 12, which is a volatilememory, stores programs loaded from the optical disk 4 or the flashmemory 17, and stores various data. The external main memory 12 is usedas a work area and a buffer area for the CPU 10. The ROM/RTC 13 includesa ROM (so-called boot ROM) incorporating a program for booting the gameapparatus body 5, and a clock circuit (RTC) for counting time. The diskdrive 14 reads, from the optical disk 4, program data, texture data andthe like, and writes the read data into an internal main memory 35described below or the external main memory 12.

The system LSI 11 includes an input/output processor (I/O processor) 31,a GPU (Graphics Processor Unit) 32, a DSP (Digital Signal Processor) 33,a VRAM (Video RAM) 34, and the internal main memory 35. These components31 to 35 are connected to each other via an internal bus (not shown).

The GPU 32, which is a part of rendering means, generates an image inaccordance with a graphics command (draw command) supplied from the CPU10. The VRAM 34 stores data (such as polygon data and texture data) usedby the GPU 32 to execute the graphics command. When an image isgenerated, the GPU 32 generates image data using the data stored in theVRAM 3. In the exemplary embodiment, the game apparatus body 5 maygenerate both a game image to be displayed on the monitor 2 and a gameimage to be displayed on the terminal apparatus 6. Hereinafter, the gameimage to be displayed on the monitor 2 may be referred to as a “monitorgame image”, and the game image to be displayed on the terminalapparatus 6 may be referred to as a “terminal game image”.

The DSP 33, serving as an audio processor, generates sound data usingsound data and sound waveform (tone quality) data stored in the internalmain memory 35 and the external main memory 12. In the exemplaryembodiment, similarly to the game images, both a game sound to be outputfrom the loudspeakers 2 a of the monitor 2 and a game sound to be outputfrom the loudspeakers of the terminal apparatus 6 may be generated.Hereinafter, the game sound to be output from the monitor 2 may bereferred to as a “monitor game sound”, and the game sound to be outputfrom the terminal apparatus 6 may be referred to as a “terminal gamesound”.

Among the image data and sound data generated by the game apparatus body5, the image data and sound data to be output to the monitor 2 are readby the AV-IC 15. The AV-IC 15 outputs the read image data to the monitor2 via an AV connector 16, and outputs the read sound data to theloudspeakers 2 a included in the monitor 2. Thereby, an image isdisplayed on the monitor 2, and a sound is output from the loudspeakers2 a.

Further, among the image data and sound data generated by the gameapparatus body 5, the image data and sound data to be output to theterminal apparatus 6 are transmitted to the terminal apparatus 6 by theI/O processor 31 or the like. Data transmission to the terminalapparatus 6 by the I/O processor 31 or the like will be described later.

The I/O processor 31 performs data reception and transmission with thecomponents connected thereto, and download of data from an externalapparatus. The I/O processor 31 is connected to the flash memory 17, thenetwork communication module 18, the controller communication module 19,an extension connector 20, a memory card connector 21, and a codec LSI27. An antenna 23 is connected to the controller communication module19. The codec LSI 27 is connected to the terminal communication module28, and an antenna 29 is connected to the terminal communication module28.

The game apparatus body 5 is connected to a network such as the Internetso as to communicate with external information processing apparatuses(for example, other game apparatuses or various servers). That is, theI/O processor 31 is connected to a network via the network communicationmodule 18 and the antenna 22 so as to communicate with externalinformation processing apparatuses connected to the network. The I/Oprocessor 31 accesses the flash memory 17 at regular intervals so as todetect for data to be transmitted to the network. When data to betransmitted is detected, the data is transmitted to the network via thenetwork communication module 18 and the antenna 22. Further, the I/Oprocessor 31 receives, via the network, the antenna 22 and the networkcommunication module 18, data transmitted from the external informationprocessing apparatuses or data downloaded from a download server, andstores the received data in the flash memory 17. The CPU 10 executes aprogram, and reads the data stored in the flash memory 17 to use thedata for execution of the program. The flash memory 17 may store notonly the data transmitted and received between the game apparatus body 5and the external information processing apparatuses, but also saved data(result data or progress data of the process) of the game played withthe game apparatus body 5. Further, the flash memory 17 may storeprograms such as a game program.

The game apparatus body 5 can receive operation data from the controller7 and/or the board-type controller 9. That is, the I/O processor 31receives, via the antenna 23 and the controller communication module 19,operation data or the like transmitted from the controller 7 and/or theboard-type controller 9, and stores (temporarily) the data in a bufferregion of the internal main memory 35 or the external main memory 12.Similarly to the external main memory 12, the internal main memory 35may store a program loaded from the optical disk 4 or a program loadedfrom the flash memory 17, and various data. The internal main memory 35may be used as a work region or buffer region of the CPU 10.

The game apparatus body 5 is capable of transmitting/receiving imagedata, sound data and the like to/from the terminal apparatus 6. Whentransmitting a game image (terminal game image) to the terminalapparatus 6, the I/O processor 31 outputs data of a game image generatedby the GPU 32 to the codec LSI 27. The codec LSI 27 performs apredetermined compression process on the image data supplied from theI/O processor 31. The terminal communication module 28 performs wirelesscommunication with the terminal apparatus 6. Accordingly, the image datacompressed by the codec LSI 27 is transmitted by the terminalcommunication module 28 to the terminal apparatus 6 via the antenna 29.In the exemplary embodiment, the codec LSI 27 compresses the image datausing a highly efficient compression technique, for example, the H.264standard. The codec LSI 27 may adopt other compression techniques. Whenthe communication rate is sufficiently high, uncompressed image data maybe transmitted. The terminal communication module 28 is, for example, aWi-Fi certified communication module. The terminal communication module28 may perform wireless communication with the terminal apparatus 6 at ahigh speed using, for example, the technique of MIMO (Multiple InputMultiple Output) adopted in the IEEE 802.11n standard, or may use othercommunication techniques.

The game apparatus body 5 transmits, to the terminal apparatus 6, sounddata as well as the image data. That is, the I/O processor 31 outputssound data generated by the DSP 33 to the terminal communication module28 via the codec LSI 27. The codec LSI 27 performs a compression processon the sound data in a similar manner to that for the image data. Anycompression technique may be adopted for the sound data. In anotherembodiment, uncompressed sound data may be transmitted. The terminalcommunication module 28 transmits the compressed image data and sounddata to the terminal apparatus 6 via the antenna 29.

The game apparatus body 5 transmits, in addition to the image data andsound data, various control data to the terminal apparatus 6, wherenecessary. The control data represent control instructions for thecomponents included in the terminal apparatus 6, such as an instructionto control on/off of a marker section (a marker section 65 shown in FIG.5), and an instruction to control image taking of a camera (a camera 66shown in FIG. 10). The I/O processor 31 transmits the control data tothe terminal apparatus 6 in response to an instruction from the CPU 5.In the exemplary embodiment, the codec LSI 27 does not perform a datacompression process on the control data. Alternatively, in anotherembodiment, the codec LSI 27 may perform a compression process on thecontrol data. The above data transmitted from the game apparatus body 5to the terminal apparatus 6 may be encrypted where necessary, or may notbe encrypted.

The game apparatus body 5 can receive various data from the terminalapparatus 6. Although details will be described later, in the exemplaryembodiment, the terminal apparatus 6 transmits operation data, imagedata, and sound data. The respective data transmitted from the terminalapparatus 6 are received by the terminal communication module 28 via theantenna 29. The image data and sound data transmitted from the terminalapparatus 6 have been subjected to a similar compression process to thatfor the image data and sound data transmitted from the game apparatusbody 5 to the terminal apparatus 6. Accordingly, these image data andsound data are transmitted from the terminal communication module 28 tothe codec LSI 27, and subjected to a decompression process by the codecLSI 27. The decompressed data are output to the I/O processor 31. On theother hand, the operation data transmitted from the terminal apparatus 6is smaller in amount than the image data and sound data, and therefore,the operation data does not need to be compressed. The operation datamay be encrypted where necessary, or may not be encrypted. Accordingly,the operation data, which has been received by the terminalcommunication module 28, is output to the I/O processor 31 via the codecLSI 27. The I/O processor 31 stores (temporarily) the data received fromthe terminal apparatus 6 in the buffer region of the internal mainmemory 35 or the external main memory 12.

The game apparatus body 5 is connectable to other devices and externalstorage media. That is, an extension connector 20 and a memory cardconnector 21 are connected to the I/O processor 31. The expansionconnector 20 is an interface connector as typified by a USB and an SCSI,and is capable of performing communication with the network, instead ofthe network communication module 18, by connecting thereto a medium suchas an external storage medium, a peripheral device such as anothercontroller, or a wired communication connector. The memory cardconnector 21 is a connector for connecting thereto an external storagemedium such as a memory card. For example, the I/O processor 31 accessesthe external storage medium via the expansion connector 20 or the memorycard connector 21 to save or read data.

The game apparatus body 5 includes (on the front main surface thereof,for example) a power button 24, a reset button 25, an insertion slot inwhich the optical disk 4 is inserted, an eject button 26 for ejectingthe optical disk 4 from the insertion slot of the game apparatus body 5,and the like. The power button 24 and the reset button 25 are connectedto the system LSI 11. When the power button 24 is turned on, therespective components of the game apparatus body 5 are supplied withpower. When the reset button 25 is pressed, the system LSI 11re-executes the boot program of the game apparatus body 5. The ejectbutton 26 is connected to the disk drive 14. When the eject button 26 ispressed, the optical disk 4 is ejected from the disk drive 14.

In another embodiment, some of the components of the game apparatus body5 may be constituted as an extension device separated from the gameapparatus body 5. At this time, the extension device may be connected tothe game apparatus body 5 via the extension connector 20. Specifically,the extension device may include, for example, the codec LSI 27, theterminal communication module 28, and the antenna 29, and may bedetachably connected to the extension connector 20. Thus, by connectingthe extension device to the game apparatus body which does not have theabove components, the game apparatus body can be made capable ofcommunicating with the terminal apparatus 6.

Next, with reference to FIGS. 3 through 5, the configuration of theterminal apparatus 6 is described. FIG. 3 is a diagram showing anexample of the external configuration of the terminal apparatus 6. Morespecifically, (a) of FIG. 3 is a front view of the terminal apparatus 6,(b) of FIG. 3 is a top view, (c) of FIG. 3 is a right side view, and (d)of FIG. 3 is a bottom view. FIG. 4 shows an example of the state where auser holds the terminal apparatus 6 with both hands.

As shown in FIG. 3, the terminal apparatus 6 includes a housing 60 whichgenerally has a horizontally long plate-like rectangular shape. Thehousing 60 is small enough to be held by the user. Therefore, the useris allowed to move the terminal apparatus 6 with hands, and change thelocation of the terminal apparatus 6.

The terminal apparatus 6 includes an LCD 61 on a front surface of thehousing 60. The LCD 61 is provided near the center of the front surfaceof the housing 60. Therefore, as shown in FIG. 4, the user, holding thehousing 60 at portions to the left and right of the LCD 61, is allowedto move the terminal apparatus 6 while viewing a screen of the LCD 61.FIG. 4 shows an example where the user holds the terminal apparatus 6horizontally (i.e., with the long side of the terminal apparatus 6oriented horizontally) by holding the housing 60 at portions to the leftand right of the LCD 61. The user, however, may hold the terminalapparatus 6 vertically (i.e., with the long side of the terminalapparatus 6 oriented vertically).

As shown in (a) of FIG. 3, the terminal apparatus 6 includes, asoperation means, a touch panel 62 on the screen of the LCD 61. In theexemplary embodiment, the touch panel 62 is, but is not limited to, aresistive film type touch panel. However, a touch panel of a given type,such as electrostatic capacitance type, may be used. The touch panel 62may be of single touch type or multiple touch type. In the exemplaryembodiment, the touch panel 62 has the same resolution (detectionaccuracy) as that of the LCD 61. The resolution of the touch panel 62and the resolution of the LCD 61, however, do not need to be the same.Although an input to the touch panel 62 is usually performed using atouch pen, in addition to the touch pen, a finger of the user may beused to perform an input to the touch panel 62. The housing 60 may havean opening for accommodating the touch pen used to perform an operationto the touch panel 62. The terminal apparatus 6 has the touch panel 62,and therefore, the user is allowed to operate the touch panel 62 whilemoving the terminal apparatus 6. That is, the user is allowed todirectly (using the touch panel 62) perform an input to the screen ofthe LCD 61 while moving the LCD 61.

As shown in FIG. 3, the terminal apparatus 6 has, as operation means,two analog sticks 63A and 63B, and a plurality of operation buttons 64Athrough 64L. The analog sticks 63A and 63B are each a device fordesignating a direction. The analog sticks 63A and 63B are eachconfigured such that a stick part thereof to be operated by a finger ofthe user is slidable or tiltable in a given direction (at a given anglein a given direction such as the upward, the downward, the leftward, therightward, or the diagonal direction) with respect to the front surfaceof the housing 60. The left analog stick 63A is provided to the left ofthe screen of the LCD 61, and the right analog stick 63B is provided tothe right of the screen of the LCD 61. Therefore, the user is allowed toperform an input for designating a direction using the analog stick 63Aor 63B with either the left or right hand. Further, as shown in FIG. 4,the analog sticks 63A and 63B are positioned so as to be operated by theuser holding the left and right portions of the terminal apparatus 6.Therefore, the user is allowed to easily operate the analog sticks 63Aand 63B when the user holds and moves the terminal apparatus 6.

The operation buttons 64A through 64L are each operation means forperforming a predetermined input. As described below, the operationbuttons 64A through 64L are positioned so as to be operated by the userholding the left and right portions of the terminal apparatus 6 (seeFIG. 4). Accordingly, the user is allowed to easily operate theoperation means when the user holds and moves the terminal apparatus 6.

As shown in (a) of FIG. 3, among the operation buttons 64A through 64L,the cross button (direction input button) 64A and the operation buttons64B through 64H are provided on the front surface of the housing 60. Theoperation buttons 64A through 64H are positioned so as to be operated bya thumb of the user (see FIG. 4).

The cross button 64A is provided to the left of the LCD 61 and beneaththe left analog stick 63A. That is, the cross button 64A is positionedso as to be operated by the left hand of the user. The cross button 64Ais cross-shaped, and is capable of indicating an upward, a downward, aleftward, or a rightward direction. The operation buttons 64B through64D are provided beneath the LCD 61. The three operation buttons 64Bthrough 64D are positioned so as to be operated by the right and lefthands of the user. The four operation buttons 64E through 64H areprovided to the right of the LCD 61 and beneath the right analog stick63B. That is, the four operation buttons 64E through 64H are positionedso as to be operated by the right hand of the user. Further, the fouroperation buttons 64E through 64H are positioned upward, downward,leftward, and rightward, respectively, with respect to a center positionof the four operation buttons. Accordingly, the terminal apparatus 6 maycause the four operation buttons 64E through 64H to function as buttonswhich allow the user to designate an upward, a downward, a leftward, ora rightward direction.

As shown in (a), (b), and (c) of FIG. 3, a first L button 641 and afirst R button 64J are provided on diagonal upper portions (an upperleft portion and an upper right portion) of the housing 60.Specifically, the first L button 641 is provided on the left end of theupper side surface of the plate-shaped housing 60 so as to protrude fromthe upper and left side surfaces. The first R button 64J is provided onthe right end of the upper side surface of the housing 60 so as toprotrude from the upper and right side surfaces. In this way, the firstL button 641 is positioned so as to be operated by the index finger ofthe left hand of the user, and the first R button 64J is positioned soas to be operated by the index finger of the right hand of the user (seeFIG. 4).

As shown in (b) and (c) of FIG. 3, leg parts 68A and 68B are provided soas to protrude from a rear surface (i.e., a surface reverse of the frontsurface on which the LCD 61 is provided) of the plate-shaped housing 60,and a second L button 64K and a second R button 64L are provided so asto protrude from the leg parts 68A and 68B, respectively. Specifically,the second L button 64K is provided at a slightly upper position on theleft side (the left side as viewed from the front surface side) of therear surface of the housing 60, and the second R button 64L is providedat a slightly upper position on the right side (the right side as viewedfrom the front-surface side) of the rear surface of the housing 60. Inother words, the second L button 64K is provided at a positionsubstantially opposite to the left analog stick 63A provided on thefront surface, and the second R button 64L is provided at a positionsubstantially opposite to the right analog stick 63B provided on thefront surface. The second L button 64K is positioned so as to beoperated by the middle finger of the left hand of the user, and thesecond R button 64L is positioned so as to be operated by the middlefinger of the right hand of the user (see FIG. 4). Further, as shown in(c) of FIG. 3, the leg parts 68A and 68B each have a surface facingobliquely upward, and the second L button 64K and the second R button64L are provided on the oblique surfaces of the leg parts 68A and 68B,respectively. Thus, the second L button 64K and the second R button 64Lhave button surfaces facing obliquely upward. It is supposed that themiddle finger of the user moves vertically when the user holds theterminal apparatus 6, and therefore, the upward facing button surfacesallow the user to easily press the second L button 64K and the second Rbutton 64L. Further, the leg parts 68A and 68B provided on the rearsurface of the housing 60 allow the user to easily hold the housing 60.Moreover, the operation buttons provided on the leg parts 68A and 68Ballow the user to easily perform operation while holding the housing 60.

In the terminal apparatus 6 shown in FIG. 3, the second L button 64K andthe second R button 64L are provided on the rear surface of the housing60. Therefore, if the terminal apparatus 6 is placed with the screen ofthe LCD 61 (the front surface of the housing 60) facing upward, thescreen of the LCD 61 may not be perfectly horizontal. Accordingly, inanother embodiment, three or more leg parts may be provided on the rearsurface of the housing 60. In this case, if the terminal apparatus 6 isplaced on a floor with the screen of the LCD 61 facing upward, the threeor more leg parts contact the floor. Thus, the terminal apparatus 6 canbe placed with the screen of the LCD 61 being horizontal. Such ahorizontal placement of the terminal apparatus 6 may be achieved byproviding detachable leg parts on the rear surface of the housing 60.

The respective operation buttons 64A through 64L are assigned functions,where necessary, in accordance with a game program. For example, thecross button 64A may be used for direction designation operation,selection operation, and the like, and the operation buttons 64E through64H may be used for determination operation, cancellation operation, andthe like.

The terminal apparatus 6 includes a power button (not shown) for turningon/off the power of the terminal apparatus 6. The terminal apparatus 6may include an operation button for turning on/off screen display of theLCD 61, an operation button for performing connection setting (pairing)with the game apparatus body 5, and an operation button for adjustingthe volume of loudspeakers (loudspeakers 607 shown in FIG. 5).

As shown in (a) of FIG. 3, the terminal apparatus 6 includes a markersection (a marker section 65 shown in FIG. 5) including a marker 65A anda marker 65B, on the front surface of the housing 60. For example, themarker section 65 is provided above the LCD 61. The markers 65A and 65Bare each constituted by one or more infrared LEDs, like the markers 8Land 8R of the marker 8. The marker section 65 is used, like the marker8, for causing the game apparatus body 5 to calculate a movement or thelike of the controller 7 with respect to the marker section 65. The gameapparatus body 5 is capable of controlling the infrared LEDs of themarker section 65 to be on or off.

The terminal apparatus 6 includes a camera 66 as imaging means. Thecamera 66 includes an image pickup element (e.g., a CCD image sensor ora CMOS image sensor) having a predetermined resolution, and a lens. Forexample, the camera 66 is provided on the front surface of the housing60. Accordingly, the camera 66 is capable of taking an image of the faceof the user holding the terminal apparatus 6. For example, the camera 66is capable of taking an image of the user playing a game while viewingthe LCD 61.

The terminal apparatus 6 has a microphone (a microphone 609 shown inFIG. 5) as sound input means. A microphone hole 60 b is provided in thefront surface of the housing 60. The microphone 609 is embedded in thehousing 60 at a position inside the microphone hole 60 b. The microphone609 detects for a sound, such as user's voice, around the terminalapparatus 6.

The terminal apparatus 6 has loudspeakers (loudspeakers 607 shown inFIG. 5) as sound output means. As shown in (d) of FIG. 3, speaker holes60 a are provided in the lower side surface of the housing 60. A soundis output through the speaker holes 60 a from the loudspeakers 607. Inthe exemplary embodiment, the terminal apparatus 6 has two loudspeakers,and the speaker holes 60 a are provided at positions corresponding to aleft loudspeaker and a right loudspeaker.

The terminal apparatus 6 includes an extension connector 67 forconnecting another device to the terminal apparatus 6. In the exemplaryembodiment, as shown in (d) of FIG. 3, the extension connector 67 isprovided in the lower side surface of the housing 60. Any device may beconnected to the extension connection 67. For example, a controller (agun-shaped controller or the like) used for a specific game or an inputdevice such as a keyboard may be connected to the extension connector67. If another device does not need to be connected, the extensionconnector 67 does not need to be provided.

In the terminal apparatus 6 shown in FIG. 3, the shapes of the operationbuttons and the housing 60, the number of the respective components, andthe positions in which the components are provided are merely examples.The shapes, numbers, and positions may be different from those describedabove.

Next, with reference to FIG. 5, the internal configuration of theterminal apparatus 6 is described. FIG. 5 is a block diagram showing anexample of the internal configuration of the terminal apparatus 6. Asshown in FIG. 5, the terminal apparatus 6 includes, in addition to thecomponents shown in FIG. 3, a touch panel controller 601, a magneticsensor 602, a gyro sensor 604, a user interface controller (UIcontroller) 605, a codec LSI 606, loudspeakers 607, a sound IC 608, amicrophone 609, a wireless module 610, an antenna 611, an infraredcommunication module 612, a flash memory 613, a power supply IC 614, abattery 615, and a vibrator 619. These electronic components are mountedon an electronic circuit board and accommodated in the housing 60.

The UI controller 605 is a circuit for controlling data input to variousinput/output sections and data output from various input/outputsections. The UI controller 605 is connected to the touch panelcontroller 601, the analog stick 63 (the analog sticks 63A and 63B), theoperation button 64 (the operation buttons 64A through 64L), the markersection 65, the magnetic sensor 602, the acceleration sensor 603, thegyro sensor 604, and the vibrator 619. Further, the UI controller 605 isconnected to the codec LSI 606 and the extension connector 67. The powersupply IC 614 is connected to the UI controller 605, so that power issupplied to the respective components through the UI controller 605. Theinternal battery 615 is connected to the power supply IC 614, so thatpower is supplied from the battery 615. Further, a battery charger 616or a cable, which is supplied with power from an external power supply,may be connected to the power supply IC 614 via a connector or the like.In this case, the terminal apparatus 6 can be supplied with power andcharged from the external power supply using the battery charger 616 orthe cable. Charging of the terminal apparatus 6 may be performed bysetting the terminal apparatus 6 on a cradle (not shown) having acharging function.

The touch panel controller 601 is a circuit which is connected to thetouch panel 62 and controls the touch panel 62. The touch panelcontroller 601 generates a predetermined form of touch position data, onthe basis of a signal from the touch panel 62, and outputs the touchposition data to the UI controller 605. The touch position datarepresents coordinates of a position at which an input is performed onan input surface of the touch panel 62. The touch panel controller 601reads a signal from the touch panel 62 and generates touch position dataevery predetermined period of time. Further, various controlinstructions on the touch panel 62 are output from the UI controller 605to the touch panel controller 601.

The analog stick 63 outputs, to the UI controller 605, stick datarepresenting a direction in which the stick part operated by a finger ofthe user slides (or tilts), and the amount of the sliding (tilting). Theoperation button 64 outputs, to the UI controller 605, operation buttondata representing an input state of each of the operation buttons 64Athrough 64L (whether or not the operation button is pressed).

The magnetic sensor 602 detects the magnitude and direction of amagnetic field to detect an orientation. Orientation data representingthe detected orientation is output to the UI controller 605. The UIcontroller 605 outputs, to the magnetic sensor 602, a controlinstruction for the magnetic sensor 602. Examples of the magnetic sensor602 include: an MI (Magnetic Impedance) sensor, a fluxgate sensor, ahall sensor, a GMR (Giant Magneto Resistance) sensor, a TMR (TunnelingMagneto Resistance) sensor, and an AMR (Anisotropic Magneto Resistance)sensor. Any sensor, however, may be adopted as long as the sensor candetect an orientation. Strictly speaking, the obtained orientation datadoes not represent an orientation in a place where a magnetic field isgenerated in addition to the geomagnetism. Even in such a case, it ispossible to calculate a change in the attitude of the terminal apparatus6 because the orientation data changes when the terminal apparatus 6moves.

The acceleration sensor 603 is provided inside the housing 60. Theacceleration sensor 603 detects the magnitudes of linear accelerationsalong three axial directions (the x-axis, y-axis, and z-axis directionsshown in (a) of FIG. 3). Specifically, in the acceleration sensor 603,the long side direction of the housing 60 is defined as the x-axisdirection (in the state where the marker section 65 is placed above theLCD 61, the right direction along the long side direction when facingthe display screen of the LCD 61 is defined as an x-axis positivedirection), the short side direction of the housing 60 is defined as they-axis direction (in the state where the marker section 65 is placedabove the LCD 61, the up direction along the short side direction whenfacing the display screen of the LCD 61 is a y-axis positive direction),and the direction orthogonal to the front surface of the housing 60 isdefined as the z-axis direction (the perspective direction of thedisplay screen of the LCD 61 is defined as a z-axis positive direction),thereby detecting the magnitudes of the linear accelerations in therespective axis directions. Acceleration data representing the detectedaccelerations is output to the UI controller 605. The UI controller 605outputs, to the acceleration sensor 603, a control instruction for theacceleration sensor 603. In the exemplary embodiment, the accelerationsensor 603 is, for example, an electrostatic capacitance type MEMSacceleration sensor. In another embodiment, however, another type ofacceleration sensor may be used. Further, the acceleration sensor 603may be an acceleration sensor for detecting the magnitude ofacceleration in one axial direction or two axial directions.

The gyro sensor 604 is provided inside the housing 60. The gyro sensor604 detects the angular velocities about the three axes (the x, y, and zaxes described above). Angular velocity data representing the detectedangular velocities is output to the UI controller 605. The UI controller605 outputs, to the gyro sensor 604, a control instruction for the gyrosensor 604. Any number and any combination of gyro sensors may be usedas long as the angular velocities about three axes are detected. Thegyro sensor 604 may be constituted by a two-axis gyro sensor and aone-axis gyro sensor. Alternatively, the gyro sensor 604 may be a gyrosensor for detecting the angular velocity about one axis or two axes.

The vibrator 619 is, for example, a vibration motor or a solenoid. Thevibrator 619 is connected to the UI controller 605. The terminalapparatus 6 is vibrated by actuating the vibrator 619 in accordance witha control instruction outputted from the UI controller 605 to thevibrator 619. The vibration of the terminal apparatus 6 is transmittedto the user's hand holding the terminal apparatus 6. Thus, a so-calledvibration-feedback game is achieved.

The UI controller 605 outputs, to the codec LSI 606, the operation dataincluding the touch position data, the stick data, the operation buttondata, the orientation data, the acceleration data, and the angularvelocity data, which have been received from the respective components.If another device is connected to the terminal apparatus 6 through theextension connector 67, data representing operation to said anotherdevice may be included in the operation data.

The codec LSI 606 is a circuit for performing a compression process ondata to be transmitted to the game apparatus body 5, and a decompressionprocess on data transmitted from the game apparatus body 5. The LCD 61,the camera 66, the sound IC 608, the wireless module 610, the flashmemory 613, and the infrared communication module 612 are connected tothe codec LSI 606. The codec LSI 606 includes a CPU 617 and an internalmemory 618. Although the terminal apparatus 6 is configured not toperform game processing, the terminal apparatus 6 may execute a programfor managing the terminal apparatus 6 or a program for communication.For example, a program stored in the flash memory 613 is loaded into theinternal memory 618 and executed by the CPU 617 when the terminalapparatus 6 is powered on, thereby starting up the terminal apparatus 6.A part of the area of the internal memory 618 is used as a VRAM for theLCD 61.

The camera 66 takes an image in accordance with an instruction from thegame apparatus body 5, and outputs data of the taken image to the codecLSI 606. The codec LSI 606 outputs, to the camera 66, a controlinstruction for the camera 66, such as an instruction to take an image.The camera 66 is also capable of taking a moving picture. That is, thecamera 66 is capable of repeatedly performing image taking, andrepeatedly outputting image data to the codec LSI 606.

The sound IC 608 is connected to the loudspeakers 607 and the microphone609. The sound IC 608 is a circuit for controlling input of sound datafrom the microphone 609 to the codec LSI 606 and output of sound datafrom the codec LSI 606 to the loudspeakers 607. Specifically, when thesound IC 608 receives sound data from the codec LSI 606, the sound IC608 performs D/A conversion on the sound data, and outputs a resultantsound signal to the loudspeakers 607 to cause the loudspeakers 607 tooutput a sound. The microphone 609 detects sound (such as user's voice)propagated to the terminal apparatus 6, and outputs a sound signalrepresenting the sound to the sound IC 608. The sound IC 608 performsA/D conversion on the sound signal from the microphone 609, and outputsa predetermined form of sound data to the codec LSI 606.

The codec LSI 606 transmits the image data from the camera 66, the sounddata from the microphone 609, and the operation data from the UIcontroller 605 (terminal operation data), to the game apparatus body 5through the wireless module 610. In the exemplary embodiment, the codecLSI 606 subjects the image data and the sound data to a compressionprocess similar to that performed by the codec LSI 27. The compressedimage data and sound data, and the terminal operation data are output tothe wireless module 610 as transmission data. The antenna 611 isconnected to the wireless module 610, and the wireless module 610transmits the transmission data to the game apparatus body 5 through theantenna 611. The wireless module 610 has the same function as theterminal communication module 28 of the game apparatus body 5. That is,the wireless module 610 has a function of connecting to a wireless LANby a method based on, for example, the IEEE 802.11n standard. The datatransmitted from the wireless module 610 may be encrypted wherenecessary, or may not be encrypted.

As described above, the transmission data transmitted from the terminalapparatus 6 to the game apparatus body 5 includes the operation data(terminal operation data), the image data, and the sound data. Ifanother device is connected to the terminal apparatus 6 through theextension connector 67, data received from said another device may beincluded in the transmission data. The infrared communication module 612performs, with another device, infrared communication based on, forexample, the IRDA standard. The codec LSI 606 may include, in thetransmission data, data received by the infrared communication, andtransmit the transmission data to the game apparatus body 5, wherenecessary.

As described above, the compressed image data and sound data aretransmitted from the game apparatus body 5 to the terminal apparatus 6.These data are received by the codec LSI 606 through the antenna 611 andthe wireless module 610. The codec LSI 606 decompresses the receivedimage data and sound data. The decompressed image data is output to theLCD 61, and an image according to the image data is displayed on the LCD61. On the other hand, the decompressed sound data is output to thesound IC 608, and a sound based on the sound data is output from theloudspeakers 607.

When control data is included in the data received from the gameapparatus body 5, the codec LSI 606 and the UI controller 605 makecontrol instructions for the respective components, according to thecontrol data. As described above, the control data represents controlinstructions for the respective components (in the exemplary embodiment,the camera 66, the touch panel controller 601, the marker section 65,the sensors 602 to 604, the vibrator 619, and the infrared communicationmodule 612) included in the terminal apparatus 6. In the exemplaryembodiment, the control instructions represented by the control data areconsidered to be instructions to start and halt (stop) the operations ofthe above components. That is, some components which are not used for agame may be halted to reduce power consumption. In this case, data fromthe halted components are not included in the transmission datatransmitted from the terminal apparatus 6 to the game apparatus body 5.The marker section 65 is constituted by infrared LEDs, and therefore,the marker section 65 is controlled by simply turning on/off the supplyof power thereto.

As described above, the terminal apparatus 6 includes the operationmeans such as the touch panel 62, the analog sticks 63, and theoperation buttons 64. Alternatively, in another embodiment, the terminalapparatus 6 may include other operation means instead of or in additionto these operation means.

The terminal apparatus 6 includes the magnetic sensor 602, theacceleration sensor 603, and the gyro sensor 604 as sensors forcalculating the movement (including the position and the attitude, or achange in the position or the attitude) of the terminal apparatus 6.Alternatively, in another embodiment, the terminal apparatus 6 mayinclude one or two of these sensors. In still another embodiment, theterminal apparatus 6 may include other sensors instead of or in additionto these sensors.

The terminal apparatus 6 includes the camera 66 and the microphone 609.Alternatively, in another embodiment, the terminal apparatus 6 may notinclude the camera 66 and the microphone 609, or may include either ofthe cameral 66 and the microphone 609.

The terminal apparatus 6 includes the marker section 65 as a componentfor calculating the positional relation between the terminal apparatus 6and the controller 7 (such as the position and/or the attitude of theterminal apparatus 6 as viewed from the controller 7). Alternatively, inanother embodiment, the terminal apparatus 6 may not include the markersection 65. In still another embodiment, the terminal apparatus 6 mayinclude other means as a component for calculating the above positionalrelation. For example, the controller 7 may include a marker section,and the terminal apparatus 6 may include an image pickup element. Inthis case, the marker 8 may include an image pickup element instead ofan infrared LED.

Next, with reference to FIGS. 6 through 8, the configuration of theboard-type controller 9 is described. FIG. 6 is a perspective viewillustrating an example of the appearance of the board-type controller 9shown in FIG. 1. As shown in FIG. 6, the board-type controller 9includes a platform 9 a on which a user stands (on which the user placestheir feet), and at least four load sensors 94 a through 94 d fordetecting a load applied to the platform 9 a. Each of the load sensors94 a through 94 d is embedded in the platform 9 a (see FIG. 7), and thepositions where the load sensors 94 a through 94 d are provided areindicated by dotted lines in FIG. 6. In the following description, thefour load sensors 94 a through 94 d may be collectively referred to as aload sensor 94.

The platform 9 a is formed in the shape of substantially a rectangularparallelepiped, and is in the shape of substantially a rectangle asviewed from the top. For example, the short side of the rectangularshape of the platform 9 a is approximately 30 cm, and the long sidethereof is approximately 50 cm. The upper surface of the platform 9 a isflat, and has a pair of planes on which the user stands with the bottomsof their feet contacting thereto. Specifically, the upper surface of theplatform 9 a has a plane (a back-left region enclosed with a double linein FIG. 6) on which the user's right foot is placed, and a plane (afront-right region enclosed with a double line in FIG. 6) on which theuser's right foot is placed. The platform 9 a has, at four cornersthereof, side surfaces each partially projecting outward in acylindrical shape.

In the platform 9 a, the four load sensors 94 a through 94 d arearranged at predetermined intervals. In the exemplary embodiment, thefour load sensors 94 a through 94 d are arranged on the periphery of theplatform 9 a, more specifically, at the four corners of the platform 9a. The intervals of the load sensors 94 a through 94 d are appropriatelyset such that the load sensors 94 a through 94 d can accurately detectthe intention of a game operation which is expressed by a manner ofapplying a load to the platform 9 a by the user.

FIG. 7 shows an example of a cross-sectional view of the board-typecontroller 9, taken along line A-A in FIG. 6, and an example of anenlarged view of a corner part where a load sensor 94 is arranged. InFIG. 7, the platform 9 a includes a support plate 90 on which the userstands, and legs 92. The load sensors 94 a through 94 d are provided inpositions where the legs 92 are provided. In the exemplary embodiment,the four legs 92 are provided at the four corners, and therefore, thefour load sensors 94 a through 94 d are also provided at thecorresponding four corners. Each leg 92 is formed by plastic molding inthe shape of substantially a cylinder with a base. Each load sensor 94is located on a spherical part 92 a provided on the base of thecorresponding leg 92. The support plate 90 is supported by the legs 92via the load sensors 94.

The support plate 90 includes an upper plate 90 a forming an uppersurface and an upper side surface portion, a lower plate 90 b forming alower surface and a lower side surface portion, and an intermediateplate 90 c provided between the upper plate 90 a and the lower plate 90b. The upper plate 90 a and the lower plate 90 b are formed by, forexample, plastic molding, and are integrated using an adhesive or thelike. The intermediate plate 90 c is, for example, formed of a singlemetal plate by press forming. The intermediate plate 90 c is fixed ontothe four load sensors 94 a through 94 d. The upper plate 90 a has, on alower surface thereof, a grid-patterned rib (not shown), and issupported by the intermediate plate 90 c via the rib. Therefore, whenthe user stands on the platform 9 a, the load is transferred to the fourlegs 92 via the support plate 90 and the load sensors 94 a through 94 d.As indicated by arrows in FIG. 7, a reaction from a floor, which isgenerated by the input load, is transferred from the legs 92 through thespherical parts 92 a, the load sensors 94 a through 94 d and theintermediate plate 90 c to the upper plate 90 a.

Each load sensor 94 is, for example, a strain gauge (strain sensor) loadcell, which is a load converter for converting an input load to anelectrical signal. In the load sensor 94, a strain-generating body 95 isdeformed according to an input load, resulting in a strain. The strainis converted into a change of electrical resistance and then convertedinto a change of voltage by a strain sensor 96 attached to thestrain-generating body 95. Therefore, the load sensor 94 outputs, froman output terminal thereof, a voltage signal indicating the input load.

The load sensor 94 may be of other types, such as a tuning fork type, astring vibration type, an electrostatic capacitance type, apiezoelectric type, a magnetostrictive type, and a gyroscopic type.

Referring back to FIG. 6, the board-type controller 9 further includes apower button 9 c. When the power button 9 c is operated (e.g., when thepower button 9 c is pressed) in the state where the board-typecontroller 9 is not activated, power is supplied to each of circuitcomponents (see FIG. 8) of the board-type controller 9. There are,however, cases in which the board-type controller 9 is powered on inaccordance with an instruction from the game apparatus body 5 andthereby supply of power to the circuit components is started. Theboard-type controller 9 may be automatically powered off when a statewhere the user does not stand thereon continues for a predeterminedperiod of time (e.g., 30 sec) or more. Further, when the power button 9c is again operated in the state where the board-type controller 9 is inthe active state, the board-type controller 9 may be powered off to stopsupply of power to the circuit components.

FIG. 8 is a block diagram showing an example of an electricalconfiguration of the board-type controller 9. In FIG. 8, flows ofsignals and data are indicated by solid arrows, and supply of power isindicated by dotted arrows.

As shown in FIG. 8, the board-type controller 9 includes a microcomputer100 for controlling the operation thereof. The microcomputer 100includes a CPU, a ROM, a RAM, and the like, which are not shown. The CPUcontrols the operation of the board-type controller 9 in accordance witha program stored in the ROM.

The power button 9 c, an AD converter 102, a DC-DC converter 104, and awireless module 106 are connected to the microcomputer 100. An antenna106 a is connected to the wireless module 106. The four load sensors 94a through 94 d are connected to the AD converter 102 via amplifiers 108.

Further, the board-type controller 9 includes a battery 110 forsupplying power to the circuit components. In another embodiment, an ACadapter may be connected to the board-type controller 9 instead of thebattery 110 so that commercial power is supplied to the circuitcomponents. In this case, instead of the DC-DC converter 104, a powercircuit, which converts alternating current into direct current andlowers and rectifies a direct-current voltage, needs to be provided inthe board-type controller 9. In the exemplary embodiment, power issupplied directly from the battery 110 to the microcomputer 100 and thewireless module 106. In other words, power is constantly supplied fromthe battery 110 to the wireless module 106 and some components (such asthe CPU) in the microcomputer 100 to detect whether or not the powerbutton 9 c is turned on and whether or not a command that instructspower-on is transmitted from the game apparatus body 5. On the otherhand, power is supplied from the battery 110 through the DC-DC converter104 to the load sensors 94 a through 94 d, the AD converter 102, and theamplifiers 108. The DC-DC converter 104 converts a voltage value ofdirect current supplied from the battery 110 into a different voltagevalue, and supplies the resultant direct current to the load sensors 94a through 94 d, the AD converter 102, and the amplifiers 108.

Supply of power to the load sensors 94 a through 94 d, the A/D converter102 and the amplifiers 108 may be performed where necessary by themicrocomputer 100 that controls the DC-DC converter 104. Specifically,when the microcomputer 100 determines that it is necessary to operatethe load sensors 94 a through 94 d to detect a load, the microcomputer100 may control the DC-DC converter 104 to supply power to the loadsensors 94 a through 94 d, the A/D converter 102 and the amplifiers 108.

When power is supplied to the load sensors 94 a through 94 d, the loadsensors 94 a through 94 d each output a signal indicating a loadinputted thereto. These signals are amplified by the respectiveamplifiers 108, and converted from analog signals into digital data bythe A/D converter 102. The digital data is input to the microcomputer100. The detected values of the load sensors 94 a through 94 d are givenidentification information of the load sensors 94 a through 94 d, sothat the load sensors 94 a through 94 d can be identified from thecorresponding detected values. Thus, the microcomputer 100 can acquirethe data indicating the detected load values of the four load sensors 94a through 94 d at the same time.

On the other hand, when the microcomputer 100 determines that it is notnecessary to operate the load sensors 94 a through 94 d, i.e., when itis not the time for load detection, the microcomputer 100 controls theDC-DC converter 104 to stop supply of power to the load sensors 94 athrough 94 d, the A/D converter 102, and the amplifiers 108. Thus, theboard-type controller 9 can operate the load sensors 94 a through 94 dto detect a load or a distance only when it is required, resulting in areduction in power consumption for load detection.

Load detection is typically required when the game apparatus body 5(FIG. 1) needs to acquire load data. For example, when game apparatusbody 5 requires load information, the game apparatus body 5 transmits aninformation acquisition command to the board-type controller 9. When themicrocomputer 100 receives the information acquisition command from thegame apparatus body 5, the microcomputer 100 controls the DC-DCconverter 104 to supply power to the load sensors 94 a through 94 d andthe like, thereby detecting a load. On the other hand, when themicrocomputer 100 does not receive a load acquisition command from thegame apparatus body 5, the microcomputer 100 controls the DC-DCconverter 104 to stop supply of power to the load sensors 94 a through94 d and the like.

The microcomputer 100 may control the DC-DC converter 104 on the basisof a determination that the time of load detection arrives atpredetermined intervals. When such periodic load detection is performed,information regarding the constant time period may be supplied andstored from the game apparatus body 5 to the microcomputer 100 of theboard-type controller 9 when the game is started, or it may bepreinstalled in the microcomputer 100.

The data indicating the detected values from the load sensors 94 athrough 94 d are transmitted as board operation data (input data) forthe board-type controller 9 from the microcomputer 100 via the radiomodule 106 and an antenna 106 b to the game apparatus body 5. Forexample, when the microcomputer 100 has performed load detectionaccording to a command from the game apparatus body 5, the microcomputer100 transmits the detected value data of the load sensors 94 a through94 d to the game apparatus body 5 on receipt of the detected value datafrom the A/D converter 102. The microcomputer 100 may transmit thedetected value data to the game apparatus body 5 at predeterminedintervals. If the interval of the data transmission is longer than theinterval of the load detection, data containing load values which havebeen detected at a plurality of detection times up to the subsequenttime of transmission may be transmitted.

The wireless module 106 is set so as to perform communication accordingto the same wireless standard (the Bluetooth, wireless LAN, and thelike) as that for the controller communication module 19 of the gameapparatus body 5. Accordingly, the CPU 10 of the game apparatus body 5is allowed to transmit an information acquisition command to theboard-type controller 9 through the controller communication module 19and the like. Thus, the board-type controller 9 is allowed to receivethe command from the game apparatus body 5 through the wireless module106 and the antenna 106 a. Further, the board-type controller 9 isallowed to transmit the board operation data including the loaddetection values (or load calculation values) of the load sensors 94 athrough 94 d to the game apparatus body 5.

For example, in a game which is performed on the basis of a simple sumof four load values detected by the four load sensors 94 a through 94 d,the user is allowed to stand at a given position with respect to thefour load sensors 94 a through 94 d of the board-type controller 9. Thatis, the user is allowed to stand on the platform 9 a at a given positionand in a given direction to play a game. In some kinds of games,however, the direction of a load value detected by each of the four loadsensors 94 viewed from the user needs to be identified. That is, apositional relation between the four load sensors 94 of the board-typecontroller 9 and the user needs to be recognized. In this case, forexample, the positional relation between the four load sensors 94 andthe user may be defined in advance, and the user may be supposed tostand on the platform 9 a in a manner which allows the predeterminedpositional relation. Typically, a positional relation in which two ofthe load sensors 94 a through 94 d are present in front of, behind, tothe right of, and to the left of the user standing in the center of theplatform 9 a, i.e., a positional relation in which the user stands inthe center of the platform 9 a of the board-type controller 9, isdefined. In this case, the platform 9 a of the board-type controller 9is rectangular in shape as viewed from the top, and the power button 9 cis provided at one side (long side) of the rectangle. Therefore, it isruled in advance that the user, using the power button 9 c as a guide,stands on the platform 9 a such that the long side at which the powerbutton 9 c is provided is located in a predetermined direction (front,rear, left or right). In this case, each of the load values detected bythe load sensors 94 a through 94 d is a load value of a predetermineddirection (front right, front left, rear right, or rear left) as viewedfrom the user. Therefore, the board-type controller 9 and the gameapparatus body 5 can find out a direction to which each detected loadvalue corresponds as viewed from the user, on the basis of theidentification information of the load sensors 94 contained in thedetected load value data, and arrangement data indicating the positionsor the directions of the load sensors 94 with respect to the user thatis set (stored) in advance. As a result, it is possible to understandthe intention of a game operation performed by the user, such as anoperating direction, for example, a forward, a backward, a leftward, ora rightward direction, or a user's foot being lifted.

Next, with reference to FIGS. 9 through 14, a description is given of anoverview of the information processing performed by the game apparatusbody 5, before descriptions are given of specific processes performed bythe game apparatus body 5. It should be noted that FIG. 9 is a diagramshowing an example of the state of a user performing an operation usingthe terminal apparatus 6 and the board-type controller 9. FIG. 10A is adiagram showing an example of an image displayed on the LCD 61 of theterminal apparatus 6. FIG. 10B is a diagram showing an example of animage displayed on the monitor 2. FIG. 11A is a diagram showing anexample of an image displayed on the LCD 61 in the case where theterminal apparatus 6 is viewed in a squarely-viewed state. FIG. 11B is adiagram showing an example where the image shown in FIG. 11A isdisplayed on the LCD 61 of the terminal apparatus 6 in thesquarely-viewed state. FIG. 12A is a diagram showing an example of animage displayed on the LCD 61 in the case where the terminal apparatus 6has been roll-rotated counterclockwise from the squarely-viewed state.FIG. 12B is a diagram showing an example where the image shown in FIG.12A is displayed on the LCD 61 of the terminal apparatus 6 roll-rotatedcounterclockwise from the squarely-viewed state. FIG. 13A is a diagramshowing an example of an image displayed on the LCD 61 in the case wherethe terminal apparatus 6 has been roll-rotated counterclockwise from thesquarely-viewed state and is viewed in a vertically-held state. FIG. 13Bis a diagram showing an example where the image displayed in FIG. 13A isdisplayed on the LCD 61 of the terminal apparatus 6 in thevertically-held state. FIG. 14 is a diagram illustrating examples of:the relationship between a terminal apparatus perspective directionprojected onto a vertical plane in real space and an operationindication direction projected onto a vertical plane in a virtual world;and a rotational movement corresponding to a terminal apparatus updirection (a roll angle).

As shown in FIG. 9, the user performs an operation using the terminalapparatus 6 and the board-type controller 9. Specifically, the useroperates the terminal apparatus 6 and the board-type controller 9 whilemounted on the board-type controller 9 and holding the terminalapparatus 6. Then, the user plays by taking action (e.g., a step actionand a leg bending and extending action) on the board-type controller 9while viewing an image displayed on the monitor 2 or an image displayedon the LCD 61 of the terminal apparatus 6, and also performing theoperation of moving the terminal apparatus 6. Then, on the LCD 61 andthe monitor 2 of the terminal apparatus 6, game images are representedsuch that a player object Po takes action in a virtual world (e.g., theaction of changing its attitude, and the action of moving its direction,and the action of changing its moving velocity) in accordance with thedirection and the attitude of the terminal apparatus 6 held by the userand the action taken by the user on the board-type controller 9, and theposition and the attitude of a virtual camera that are set in thevirtual world are changed in accordance with the position and theattitude of the player object Po.

As shown in FIG. 10A, on the LCD 61 of the terminal apparatus 6, aplayer object Po moving in a virtual world is displayed. In the exampleshown in FIG. 10A, a virtual camera is set behind and close to theplayer object Po swimming under the sea, and the state of the virtualworld viewed from the virtual camera is displayed together with theplayer object Po. The virtual world viewed from behind and close to theplayer object Po is thus displayed on the LCD 61, whereby it is possibleto provide the user, holding the terminal apparatus 6, with a sense ofpresence in the virtual world, and it is also possible to allow the userto understand the moving direction and the moving velocity of the playerobject Po in an intuitive manner. Further, the attitude and the movingdirection of the player object Po change by changing (pitching) theattitude of the terminal apparatus 6 upward and downward, rotationallymoving (yawing) the terminal apparatus 6 so as to change the directionof the terminal apparatus 6, and rotating (rolling) the terminalapparatus 6 about the perspective direction of the display screen of theterminal apparatus 6 (the z-axis direction). Also the position and theattitude of the virtual camera change in accordance with the change inthe attitude of the player object Po. For example, the attitude and thedirection of the terminal apparatus 6 may be brought into conjunctionwith the attitude and the direction of the virtual camera, whereby theuser can enjoy a feeling as if peeping at the virtual world through theLCD 61 of the terminal apparatus 6.

In addition, as shown in FIG. 10B, also on the monitor 2, the samevirtual world as the virtual world displayed on the LCD 61 is displayed.In the example shown in FIG. 10B, the state of the virtual world viewedfrom a distant bird's-eye view is displayed together with the playerobject Po swimming under the sea. In the example shown in FIG. 10B, amovement path Lp of the player object Po having swum under the sea isdisplayed on the virtual world. The state of the virtual world in whichthe player object Po is viewed from a distant bird's-eye view is thusdisplayed on the monitor 2, whereby the user can easily understand thecircumstance of the player object Po, and another person viewing thestate of the user playing can also enjoy viewing the state of the playerobject Po moving in the virtual world.

As an example, when the user has taken action so as to step on theboard-type controller 9, the player object Po takes the action ofswimming under the sea by flutter-kicking at a velocity based on thestep action. Further, when the user has taken action so as to performleg bending and extending on the board-type controller 9, the playerobject Po takes the action of swimming under the sea by dolphin-kickingat a velocity based on the leg bending and extending action. By thustaking action on the board-type controller 9, the user can change theswimming style and the moving velocity of the player object Po.

For example, as described above, the board-type controller 9 outputsdetected load values based on the action taken by the user on theboard-type controller 9. Then, the use of the detected load values makesit possible to calculate the total load applied to the board-typecontroller 9 and the position of the center of gravity of the loadapplied to the board-type controller 9. Further, the use of the totalload makes it possible to estimate the type of action that is beingtaken by the user on the board-type controller 9. The swimming style andthe moving velocity of the player object Po are set in accordance withthe action of the user thus estimated on the board-type controller 9.

In addition, in accordance with the direction and the attitude of theterminal apparatus 6 held by the user, the attitude and the movingdirection of the player object Po swimming under the sea change. As anexample, as a result of the user directing (i.e., pitching and yawing)the terminal apparatus 6 upward, downward, leftward, and rightward, inaccordance with the change in the direction of the terminal apparatus 6,the attitude of the player object Po swimming under the sea (thedirection in which the player object Po is directed in the virtualworld) changes in conjunction with the change in the direction of theterminal apparatus 6, and also the swimming direction of the playerobject Po changes. Specifically, when the user has changed the directionof the terminal apparatus 6 so as to direct the back surface of theterminal apparatus 6 upward (i.e., pitch the terminal apparatus 6 in theelevation direction), the attitude of the player object Po swimming ischanged such that the player object Po is directed in the direction ofrising in the direction of the sea surface, and also the swimmingdirection of the player object Po is changed such that the player objectPo rises in the direction of the sea surface. Further, when the user haschanged the direction of the terminal apparatus 6 so as to direct theback surface of the terminal apparatus 6 leftward (i.e., yaw theterminal apparatus 6 to the left), the attitude of the player object Poswimming is changed to the left of the player object Po, and also theswimming direction of the player object Po is changed to the left. Theuser can thus change the attitude and the moving direction of the playerobject Po on the basis of the direction and the attitude of the terminalapparatus 6 held by the user. For example, the attitude and thedirection of the terminal apparatus 6 may be brought into conjunctionwith the attitude and the moving direction of the player object Po,whereby the user can perform, using the terminal apparatus 6, anoperation having verisimilitude as if the user themselves were theplayer object Po. Further, as described above, a virtual camera is setbehind and close to the player object Po swimming under the sea. Thus,in accordance with changes in the attitude and the moving direction ofthe player object Po, also the attitude and the position of the virtualcamera change. For example, the attitude and the direction of theterminal apparatus 6 may be brought into conjunction with the attitudeand the direction of the virtual camera, whereby the user can enjoy afeeling as if peeping at the virtual world through the LCD 61 of theterminal apparatus 6.

In addition, when the user has rotated (i.e., rolled) the terminalapparatus 6, held by the user, about the perspective direction of theLCD 61 (the z-axis direction), the attitude of the player object Poswimming under the sea changes in accordance with the direction and theattitude of the terminal apparatus 6.

As shown in FIGS. 11A and 11B, when the terminal apparatus 6 is viewedin a squarely-viewed state, control is performed such that the playerobject Po swims under the sea so as to have such an attitude that thefront surface of the torso of the player object Po is directed downwardin the virtual world (i.e., the back surface of the torso is directedupward in the virtual world). Then, the player object Po and the virtualworld are displayed on the LCD 61 of the terminal apparatus 6 such thatthe up-down direction in the virtual world where the player object Po isplaced corresponds to the up-down direction of the LCD 61. Here, thesquarely-viewed state of the terminal apparatus 6 means the state wherethe user holds the terminal apparatus 6 horizontally (i.e., with thelong side of the terminal apparatus 6 oriented horizontally) by holdingthe housing 60 at portions to the left and right of the LCD 61 with themarker section 65 placed above the LCD 61, and also means the statewhere the x-axis positive direction is directed to the right of the userand along a horizontal direction in real space. Typically, thesquarely-viewed state of the terminal apparatus 6 according to theexemplary embodiment is the state where: the x-axis direction isdirected in a horizontal direction in real space; the y-axis positivedirection is directed upward in real space (i.e., directed in adirection included in a predetermined elevation range about the verticaldirection in real space); and the z-axis positive direction is directedin a direction included in a predetermined angular range about ahorizontal direction in real space.

Then, when the terminal apparatus 6 in the squarely-viewed state hasbeen rotated counterclockwise about the perspective direction of the LCD61 (the z-axis direction) (i.e., has been rolled in the direction Ashown in FIG. 11B), an image as shown in FIGS. 12A and 12B is displayedon the LCD 61. As shown in FIGS. 12A and 12B, when the terminalapparatus 6 has been roll-rotated counterclockwise from thesquarely-viewed state, control is performed such that the player objectPo swims under the sea so as to have such an attitude that the torso ofthe player object Po rotates by the angle by which the terminalapparatus 6 has roll-rotated (a roll angle; e.g., the angle by which they-axis has rotationally moved about the z-axis), and in the direction ofthe roll rotation (a roll direction; e.g., the direction in which they-axis has rotationally moved about the z-axis).

As an example, as shown in FIG. 14, when the terminal apparatus 6 hasroll-rotated in the direction A and by a roll angle θ (i.e., the y-axishas roll-rotated about the z-axis in the direction A and by the angleθ), the attitude of the player object Po is controlled such that theplayer object Po rolls in the virtual world about the moving directionof the player object Po in the direction A and by the roll angle θ(e.g., the direction of the back surface of the torso roll-rotates aboutthe moving direction in the direction A and by the angle θ). Then, theplayer object Po and the virtual world are displayed on the LCD 61 ofthe terminal apparatus 6 such that the up-down direction in the virtualworld where the player object Po is placed rotates from the up-downdirection of the LCD 61 in the direction opposite to the roll directionof the terminal apparatus 6 and by the roll angle θ. Thus, as is clearby comparing FIGS. 11A and 12A, the player object Po rolls in thevirtual world in the direction A and by the roll angle θ; however, thevirtual world is displayed on the LCD 61 so as to rotate in thedirection opposite to the direction A and by the roll angle θ. Thisresults in causing the player object Po to be displayed in the samestate on the LCD 61, and causing the virtual world except for the playerobject Po to be displayed so as to rotate in the direction opposite tothe direction A. Further, as is clear by comparing FIGS. 11B and 12B,the terminal apparatus 6 rotates from the squarely-viewed state in thedirection A and by the roll angle θ; however, the virtual world exceptfor the player object Po is displayed on the LCD 61 so as to rotate inthe direction opposite to the direction A and by the same roll angle θ.This results in causing the virtual world to be displayed in the samedirection relative to real space, and causing the player object Po to bedisplayed so as to rotate in the direction A relative to the virtualworld and real space.

When the image of the virtual world shown in FIGS. 11A and 11B, wherethe player object Po is placed, is changed to an image as shown in FIGS.12A and 12B in accordance with the roll rotation of the terminalapparatus 6, it is possible to cause the virtual camera placed behindand close to the player object Po to roll-rotate about the direction ofthe line of sight of the virtual camera in a similar manner to theplayer object Po. For example, as shown in FIG. 14, when the terminalapparatus 6 has roll-rotated in the direction A and by the roll angle θ,the attitude of the virtual camera may be controlled such that thevirtual camera placed behind and close to the player object Po rollsabout the direction of the line of sight of the virtual camera (theZ-axis positive direction shown in the figures) in the direction A andby the roll angle θ. This causes the virtual camera to move with thesame roll rotation as that of the player object Po, and results incausing the player object Po to be displayed in the same state on theLCD 61, and causing the virtual world except for the player object Po tobe displayed so as to rotate in the direction opposite to the directionA.

When the terminal apparatus 6 roll-rotated in the direction A has beenroll-rotated further in the direction A so as to enter a vertically-heldstate (i.e., has been rotationally moved about the z-axis further in thedirection A such that the y-axis is horizontal), an image as shown inFIGS. 13A and 13B is displayed on the LCD 61. As shown in FIGS. 13A and13B, when the terminal apparatus 6 has been roll-rotatedcounterclockwise from the squarely-viewed state so as to enter thevertically-held state, control is performed such that the player objectPo swims under the sea so as to have such an attitude that the torso ofthe player object Po rotates by the roll angle by which the terminalapparatus 6 has further roll-rotated and in the roll direction (e.g.,such an attitude that the direction of the back surface of the torsorotationally moves about the moving direction further in the direction Aso as to be horizontal). Then, the player object Po and the virtualworld are displayed on the LCD 61 of the terminal apparatus 6 such thatthe direction from the top to bottom of the virtual world where theplayer object Po is placed corresponds to the direction from the rightto left of the LCD 61. As is clear by comparing FIGS. 11A and 13A, thiscauses the player object Po to roll in the virtual world in thedirection A and by a roll angle of 90°; however, the virtual world isdisplayed on the LCD 61 so as to rotate in the direction opposite to thedirection A and by the same roll angle of 90°. This results in causingthe player object Po to be displayed in the same state on the LCD 61,and causing the virtual world except for the player object Po to bedisplayed so as to rotate 90° in the direction opposite to the directionA. Further, as is clear by comparing FIGS. 11B and 13B, the terminalapparatus 6 rotates from the squarely-viewed state to thevertically-held state in the direction A and by a roll angle of 90°;however, the virtual world except for the player object Po is displayedon the LCD 61 so as to rotate in the direction opposite to the directionA and by a roll angle of 90°. This results in causing the virtual worldto be displayed in the same direction relative to real space, andcausing the player object Po to be displayed so as to rotate 90° in thedirection A relative to the virtual world and real space.

As a result of the user thus roll-rotating the terminal apparatus 6, itis possible to cause the player object Po to roll-rotate in the virtualworld. The roll angle and the roll direction of the terminal apparatus 6are brought into conjunction with the roll angle and the roll directionof the player object Po, whereby the user can perform, using theterminal apparatus 6, an operation having verisimilitude as if the userthemselves were the player object Po. Further, as described above, thevirtual camera is set behind and close to the player object Po swimmingunder the sea, and the player object Po roll-rotates in a similarmanner, whereby the user can enjoy a feeling as if peeping at thevirtual world using the LCD 61 of the terminal apparatus 6 as a peepwindow.

It should be noted that if the attitudes of the virtual camera and theplayer object Po are controlled on the basis of the roll of the terminalapparatus 6, taking into account also the control of the attitudesthereof corresponding to the pitch and yaw of terminal apparatus 6 asdescribed above, the attitude of the virtual camera in the virtual worldmay be controlled so as to be, as a result, the same as the attitude ofthe terminal apparatus 6 in real space. That is, the attitude of thevirtual camera is controlled such that: the z-axis positive direction ofthe terminal apparatus 6 in real space (a terminal apparatus perspectivedirection) corresponds to the Z-axis positive direction (the directionof the line of sight) of the virtual camera in the virtual world; they-axis positive direction of the terminal apparatus 6 in real space (aterminal apparatus up direction) corresponds to the Y-axis positivedirection of the virtual camera in the virtual world (a virtual cameraup direction); and the x-axis positive direction of the terminalapparatus 6 in real space (a terminal apparatus right direction)corresponds to the X-axis positive direction of the virtual camera inthe virtual world (a virtual camera right direction). Then, thepositional relationship (including the roll directions) between theplayer object Po and the virtual camera is fixed, and the position ofthe virtual camera is set on the basis of the position of the playerobject Po that is moving in the virtual world. The virtual camera andthe player object Po are thus set in the virtual world, whereby imagesof the virtual world as shown in FIGS. 11A, 12A, and 13A can bedisplayed on the LCD 61 of the terminal apparatus 6 having attitudes asshown in FIGS. 11B, 12B, and 13B.

For example, acceleration data or angular velocity data based on achange in the attitude of the terminal apparatus 6 is output from theterminal apparatus 6. Then, the direction of the gravitationalacceleration applied to the terminal apparatus 6 can be calculated usingthe acceleration indicated by the acceleration data. This makes itpossible to estimate the attitude of the terminal apparatus 6 withrespect to the vertical direction in real space, that is, the x-axis,y-axis, and z-axis directions of the terminal apparatus 6 with respectto the vertical direction. Further, the use of the angular velocityand/or the dynamic acceleration applied to the terminal apparatus 6using the angular velocity indicated by the angular velocity data and/orthe acceleration indicated by the acceleration data, makes it possibleto estimate a change in the attitude of the terminal apparatus from itsinitial attitude in real space (i.e., changes in the x-axis, y-axis, andz-axis directions) using the angular velocity and/or the dynamicacceleration. In accordance with the thus estimated change in theattitude of the terminal apparatus 6 (changes in the x, y, and z axes),the attitude and the moving direction of the player object Po and theattitude and the position of the virtual camera are set.

Next, the information processing performed by the game system 1 isdescribed in detail. First, with reference to FIG. 15, main data used inthe information processing is described. It should be noted that FIG. 15is a diagram showing an example of main data and programs that arestored in the external main memory 12 and/or the internal main memory 35(hereinafter, these two main memories will be collectively referred tosimply as a “main memory”) of the game apparatus body 5.

As shown in FIG. 15, in a data storage area of the main memory, thefollowing are stored: board operation data Da; terminal operation dataDb; load value data Dc; position-of-center-of-gravity data Dd; terminalapparatus direction/attitude data De; operation direction data Df;action/attitude data Dg; movement vector data Dh; position data Di;virtual camera data Dj; movement path data Dk; movement mode flag dataDm; image data Dn; and the like. It should be noted that the main memoryappropriately stores, as well as the data shown in FIG. 15, data usedfor the game processing, such as image data of various objects displayedon the monitor 2 and the LCD 61, and sound data used for the game.Further, in a program storage area of the main memory, various programsPa included in the information processing program are stored.

As the board operation data Da, a series of operation information (boardoperation data) transmitted as transmission data from the board-typecontroller 9 is stored, and updated to the latest board operation data.For example, the board operation data Da includes load data Da1 and thelike. The load data Da1 is data indicating load values detected by theload sensors 94 a through 94 d of the board-type controller 9.

As the terminal operation data Db, a series of operation information(terminal operation data) transmitted as transmission data from theterminal apparatus 6 is stored, and updated to the latest terminaloperation data. For example, the terminal operation data Db includesacceleration data Db1, angular velocity data Db2, and the like. Theacceleration data Db1 is data indicating an acceleration (anacceleration vector) detected by the acceleration sensor 603. Forexample, the acceleration data Db1 represents a three-dimensionalacceleration vector whose components are accelerations in the threeaxial (x-axis, y-axis, and z-axis) directions shown in FIG. 3. Inanother embodiment, the acceleration data Db1 may representaccelerations in given one or more directions. The angular velocity dataDb2 is data representing an angular velocity detected by the gyro sensor604. For example, the angular velocity data Db2 represents angularvelocities about the three axes (x-axis, y-axis, and z-axis) shown inFIG. 3. In another example, the angular velocity data Db2 may representangular velocities about given one or more axes.

It should be noted that the game apparatus body 5 sequentially receivesthe data (e.g., the data indicating the detected load values, theacceleration, and the angular velocity) included in the operationinformation transmitted from the controller 7, the board-type controller9, and the terminal apparatus 6 at predetermined intervals (e.g., atintervals of 1/200 seconds). For example, the received data issequentially stored in the main memory by the I/O processor 31. In aprocessing flow described later, the CPU 10 reads the latest boardoperation data and the latest terminal operation data from the mainmemory every frame period (e.g., 1/60 seconds), to thereby update theboard operation data Da and the terminal operation data Db.

In addition, the operation information transmitted from the controller7, the board-type controller 9, and the terminal apparatus 6 at thepredetermined intervals may be temporarily stored in the buffer (notshown) included in the controller communication module 19 or theterminal communication module 28. In this case, the data stored in thebuffer is read every frame period, and the board operation data Da(e.g., the load data Da1) or the terminal operation data Db (e.g., theacceleration data Db1 and the angular velocity data Db2) in the mainmemory is updated for use. At this time, the cycle of receiving theoperation information is different from the processing cycle, andtherefore, a plurality of pieces of information received at a pluralityof times are stored in the buffer. The processing may be performed usingonly the latest operation information among the plurality of pieces ofoperation information received at the plurality of times.

The load value data Dc is an aggregate of data indicating the loadvalues detected by the board-type controller 9. For example, the loadvalue data Dc is an aggregate of data indicating the sum of the loadvalues (the total load value) detected by the load sensors 94 a through94 d. Specifically, the load value data Dc is an array of dataindicating the total load values within a predetermined period that arechronologically calculated, and the data indicating the total loadvalues is chronologically stored in the elements of the array.

The position-of-center-of-gravity data Dd is an aggregate of dataindicating the position of the center of gravity of the load applied tothe board-type controller 9. For example, theposition-of-center-of-gravity data Dd is an aggregate of data indicatingthe position of the center of gravity calculated, using a predeterminedformula, from the load values detected by the load sensors 94 a through94 d. Specifically, the position-of-center-of-gravity data Dd is anarray of data indicating the position of the center of gravity within apredetermined period that are chronologically calculated, and the dataindicating the position of the center of gravity is chronologicallystored in the elements of the array.

The terminal apparatus direction/attitude data De includes real spacereference direction data De1, terminal apparatus perspective directiondata De2, terminal apparatus up direction data De3, and the like. Thereal space reference direction data De1 is data indicating a referencedirection (the attitude; a real space reference direction) of theterminal apparatus 6 in real space. The terminal apparatus perspectivedirection data De2 is data indicating the current perspective directionof the terminal apparatus 6 in real space (the terminal apparatusperspective direction). The terminal apparatus up direction data De3 isdata indicating the current up direction of the terminal apparatus 6 inreal space (the terminal apparatus up direction). For example, the realspace reference direction data De1, the terminal apparatus perspectivedirection data De2, and the terminal apparatus up direction data De3 arecalculated on the basis of the acceleration data Db1 and the angularvelocity data Db2 that are included in the terminal operation data Db.The method of calculating the real space reference direction, theterminal apparatus perspective direction, and the terminal apparatus updirection will be described later.

The operation direction data Df includes virtual world referencedirection data Df1, operation indication direction data Df2, operationup direction data Df3, and the like. The virtual world referencedirection data Df1 is data indicating the virtual world referencedirection set in the virtual world. The operation indication directiondata Df2 is data indicating an operation indication direction currentlyindicated in the virtual world by the user. The operation up directiondata Df3 is data indicating an operation up direction currentlyindicated in the virtual world by the user. The method of calculatingthe virtual world reference direction, the operation indicationdirection, and the operation up direction will be described later.

The action/attitude data Dg is data indicating the action and theattitude of the player object Po in the virtual world. The movementvector data Dh is data indicating the moving velocity and the movingdirection of the player object Po in the virtual world (e.g., dataindicating the movement vector in the virtual world). The position dataDi is data indicating the position of the player object Po in thevirtual world. The virtual camera data Dj is data concerning virtualcameras set in the virtual world. For example, the virtual camera dataDj includes data concerning a first virtual camera for generating a gameimage to be displayed on the LCD 61 of the terminal apparatus 6, anddata concerning a second virtual camera for generating a game image tobe displayed on the monitor 2. The movement path data Dk is dataindicating the path of the player object Po having moved in the virtualworld.

The movement mode flag data Dm is data indicating an action mode flagthat is set in accordance with the action (e.g., swimming style) set forthe player object Po. As an example, when the user has taken action soas to perform leg bending and extending on the board-type controller 9,the action of the player object Po is set to the action of swimmingunder the sea by dolphin-kicking, and the action mode flag is set to on.

The image data Dn includes player object data Dn1, background image dataDn2, and the like. The player object data Dn1 is data for placing theplayer object Po in the virtual world, to generate a game image. Thebackground image data Dn2 is data for placing a background in thevirtual world to generate a game image.

Next, with reference to FIGS. 16 through 20, the information processingperformed by the game apparatus body 5 is described in detail. It shouldbe noted that FIG. 16 is a flow chart showing an example of theinformation processing performed by the game apparatus body 5. FIG. 17is a subroutine flow chart showing an example of a game control processin step 44 in FIG. 16. FIG. 18 is a subroutine flow chart showing anexample of a player object setting process in step 83 in FIG. 17. FIG.19 is a subroutine flow chart showing an example of a player objectaction setting process in step 127 in FIG. 18. FIG. 20 is a diagramillustrating an example of the relationships among: the real spacereference direction and the terminal apparatus perspective directionthat are projected onto a horizontal plane in real space; and thevirtual world reference direction and the operation indication directionthat are projected onto a horizontal plane in the virtual world. Here,in the flow charts shown in FIGS. 16 through 19, descriptions are givenmainly of, among the processes of the processing, a process where theplayer object Po is displayed so as to move in accordance with theoperation performed by the user using the terminal apparatus 6 and theboard-type controller 9, while detailed descriptions of the otherprocesses not directly related to the exemplary embodiment are omitted.Further, in FIGS. 16 through 19, each step performed by the CPU 10 isabbreviated as “S”.

When the game apparatus body 5 has been powered on, the CPU 10 of thegame apparatus body 5 executes a boot program stored in the ROM/RTC 13to initialize each unit such as the main memory. Then, the informationprocessing program stored in the optical disk 4 is loaded to the mainmemory, and the CPU 10 starts to execute the program. The flow chartsshown in FIGS. 16 through 19 show processes to be performed after theabove processes are completed.

Referring to FIG. 16, the CPU 10 performs an initialization process(step 40), and proceeds to the subsequent step. For example, in theinitialization process in step 40, the CPU 10 constructs the virtualworld, places the player object Po and objects in the virtual world atinitial positions, and sets the initial values of various parametersused for the game processing.

Next, the CPU 10 sets a reference direction on the basis of datatransmitted from the terminal apparatus 6 (step 41), and proceeds to thesubsequent step. A description is given below of an example where theCPU 10 sets the reference direction.

The terminal apparatus 6 repeatedly transmits data as described above tothe game apparatus body 5. In the game apparatus body 5, the terminalcommunication module 28 sequentially receives the data described above,and the I/O processor 31 sequentially stores terminal operation data,camera image data, and microphone sound data in the main memory. In step41 described above, the CPU 10 reads the most recent terminal operationdata from the main memory, to thereby update the acceleration data Db1and the angular velocity data Db2.

Next, the CPU 10 calculates the direction and the attitude of theterminal apparatus 6 in real space. For example, the CPU 10 calculates,as the reference direction (initial attitude) in real space, the currentdirection and attitude of the terminal apparatus 6 on the basis of theacceleration indicated by the acceleration data Db1 and the angularvelocity indicated by the angular velocity data Db2, to thereby updatethe real space reference direction data De1 using data indicating thecalculated reference direction of the terminal apparatus 6. For example,the CPU 10 can calculate the amount of rotation (the amount of change inthe direction) of the terminal apparatus 6 in real space per unit time,using the angular velocity indicated by the angular velocity data Db2.Further, in the state where the terminal apparatus 6 is substantiallystationary (in a static state) in real space, the acceleration appliedto the terminal apparatus 6 is the gravitational acceleration. Thismakes it possible to calculate the direction of gravity applied to theterminal apparatus 6 (i.e., the attitude of the terminal apparatus 6with respect to the vertical direction in real space), using theacceleration indicated by the acceleration data Db1. This enables theCPU 10 to calculate the initial attitude of the terminal apparatus 6 onthe basis of the acceleration indicated by the acceleration data Db1 andthe angular velocity indicated by the angular velocity data Db2. Itshould be noted that in the following descriptions, when step 41described above is performed, the real space reference direction is seton the basis of the direction in which the back surface of the terminalapparatus 6 is directed in real space (the z-axis positive directionshown in FIG. 3, i.e., the terminal apparatus perspective direction).

It should be noted that the initial attitude of the terminal apparatus 6may be calculated on the basis of the acceleration indicated by theacceleration data Db1, or may be calculated on the basis of thedirection of magnetism detected by the magnetic sensor 602.Alternatively, as a result of the user performing a predeterminedoperation in the state where the terminal apparatus 6 is in a specificattitude, the specific attitude when the predetermined operation hasbeen performed may be used as the initial attitude. It should be notedthat the initial attitude needs to be calculated if the attitude of theterminal apparatus 6 is calculated as an absolute attitude with respectto a predetermined direction in real space. Timing may be set such thatthe setting of the initial attitude, that is, step 41 described above,is performed at the start of the game, or is performed in accordancewith a predetermined operation performed by the user using the terminalapparatus 6 (e.g., the operation of pressing a predetermined operationbutton 64).

In addition, in step 41 described above, the CPU calculates, as areference direction (initial attitude) in the virtual world, the currentdirection of the player object Po in which it is directed in the virtualworld (e.g., the direction that serves as the moving direction of theplayer object Po if it moves as it is), to thereby update the virtualworld reference direction data Df1 using data indicating the calculatedreference direction in the virtual world.

Subsequent to step 41 described above, the process in step 42 isperformed. Thereafter, the processing loop of a series of processes 42through 51 is performed every predetermined period (one frame period)and repeated.

In step 42, the CPU 10 acquires board operation data transmitted fromthe board-type controller 9, and proceeds to the subsequent step. Here,the board-type controller 9 repeatedly transmits the board operationdata to the game apparatus body 5. Accordingly, in the game apparatusbody 5, the controller communication module 19 sequentially receives theboard operation data, and the I/O processor 31 sequentially stores thereceived board operation data in the main memory. The interval oftransmission of the board operation data from the board-type controller9 may be shorter than the game processing period (one frame period), andit is 1/200 seconds, for example. In step 42, the CPU 10 reads thelatest board operation data from the main memory, to thereby update theboard operation data Da. The board operation data includes dataindicating identification information of the load sensors 94 a through94 d, and data indicating the load values detected by the load sensors94 a through 94 d. The load data Da1 is updated using the dataidentified by the identification information.

Next, the CPU 10 acquires various data transmitted from the terminalapparatus 6 (step 43), and proceeds to the subsequent step. The terminalapparatus 6 repeatedly transmits the data to the game apparatus body 5.Accordingly, in the game apparatus body 5, the terminal communicationmodule 28 sequentially receives the data, and the codec LSI 27sequentially performs a decompression process on the camera image dataand the microphone sound data. Then, the I/O processor 31 sequentiallystores the terminal operation data, the camera image data, and themicrophone sound data in the main memory. In step 43 described above,the CPU 10 reads the latest terminal operation data from the mainmemory, to thereby update the acceleration data Db1 and the angularvelocity data Db2.

Next, the CPU 10 performs a game control process (step 44), and proceedsto the subsequent step. The game control process is the process of, forexample, causing the player object Po in the virtual world to move inaccordance with a game operation performed by the user, to therebyadvance the game. In this exemplary game, the user is allowed to playvarious games using the terminal apparatus 6 and the board-typecontroller 9. With reference to FIG. 17, a description is given below ofthe game control process in step 44 described above.

In FIG. 17, the CPU 10 calculates a load value and the position of thecenter of gravity (step 81), and proceeds to the subsequent step. Forexample, the CPU 10 calculates a total load value by summing up thedetected load values indicated by the load data Da1, to thereby updatethe latest data in the chronological data array of the load value dataDc, using the data indicating the calculated total load value.Specifically, the load data Da1 indicates the latest load valuesdetected by the load sensors 94 a through 94 d, and therefore, the totalload value is calculated by summing up the detected load values.Further, the CPU 10 calculates the position of the center of gravityusing the detected load values indicated by the load data Da1, tothereby update the latest data in the chronological data array of theposition-of-center-of-gravity data Dd using data indicating the positionof the center of gravity. A description is given below of an example ofthe method of calculating the position of the center of gravity.

The position of the center of gravity described above is the position ofthe center of gravity of the load applied to the platform 9 a of theboard-type controller 9, and is defined by the load values detected bythe load sensors 94 a through 94 d (see FIG. 6). For example, theposition of the center of gravity is represented by a coordinate valuebased on a predetermined coordinate system corresponding to positions onthe platform 9 a of the board-type controller 9 (e.g., an X1-Y1coordinate system where: the origin is the center of the platform 9 a;an X1-axis direction is the long side direction of the platform 9 a; anda Y1-axis direction is the short side direction). Then, in the casewhere: the load value detected by the load sensor 94 a is a; the loadvalue detected by the load sensor 94 b is b; the load value detected bythe load sensor 94 c is c; and the load value detected by the loadsensor 94 d is d, the X1-axis coordinate value (X1) and the Y1-axiscoordinate value (Y1) of the center of gravity can be calculated usingthe following formula.

X1=((a+c)−(b+d))×m

Y1=((c+d)−(a+b))×n

(where m and n are constants defined in advance)

The thus calculated total load value and position of the center ofgravity change in accordance with the action taken by the user and theshifting of their weight (the attitude) on the board-type controller 9.As an example, when the user has repeatedly stepped on the board-typecontroller 9, the total load value varies in accordance with the stepcycle, and the position of the center of gravity changes so as to shift,in accordance with the step cycle, back and forth between the positionon which the user places their left foot and the position on which theuser places their right foot. As another example, when the user hasrepeatedly performed leg bending and extending on the board-typecontroller 9, the total load value varies in accordance with the legbending and extending cycle. Changes in the position of the center ofgravity, however, are relatively reduced.

Next, the CPU 10 calculates a change in the direction and the attitudeof the terminal apparatus 6 (step 82), and proceeds to the subsequentstep. For example, the CPU 10 calculates the terminal apparatusperspective direction (the z-axis positive direction) and the terminalapparatus up direction (the y-axis positive direction) of the terminalapparatus 6 in real space on the basis of the acceleration indicated bythe acceleration data Db1 and the angular velocity indicated by theangular velocity data Db2, to thereby update the terminal apparatusperspective direction data De2 and the terminal apparatus up directiondata De3 using data indicating the calculated terminal apparatusperspective direction and terminal apparatus up direction of theterminal apparatus 6.

Here, the CPU 10 can calculate the amount of rotation (the amount ofchange in the direction) of the terminal apparatus 6 in real space perunit time, using the angular velocity indicated by the angular velocitydata Db2. Further, in the state where the terminal apparatus 6 issubstantially stationary (in a static state) in real space, theacceleration applied to the terminal apparatus 6 is the gravitationalacceleration. This makes it possible to calculate the direction ofgravity applied to the terminal apparatus 6 (i.e., the attitude of theterminal apparatus 6 with respect to the vertical direction in realspace, and the x-axis, y-axis, and z-axis directions with respect to thevertical direction), using the acceleration indicated by theacceleration data Db1. This enables the CPU 10 to calculate a change inthe direction and the attitude of the terminal apparatus 6 on the basisof the acceleration indicated by the acceleration data Db1 and theangular velocity indicated by the angular velocity data Db2.

It should be noted that in the exemplary embodiment, a change in thedirection and the attitude of the terminal apparatus 6 are calculated onthe basis of the data indicating the acceleration and the angularvelocity that are detected by the terminal apparatus 6. Alternatively,in another embodiment, a change in the direction and the attitude of theterminal apparatus 6 may be calculated using any one piece of data orthree or more pieces of data. For example, the magnetic sensor 602included in the terminal apparatus 6 detects a geomagnetism applied tothe terminal apparatus 6. This makes it possible to calculate apredetermined orientation with respect to the terminal apparatus 6(i.e., the attitude of the terminal apparatus 6 with respect to thepredetermined orientation) on the basis of the direction of thegeomagnetism applied to the terminal apparatus 6. Even when a magneticfield is generated in addition to the geomagnetism in the real spacewhere the terminal apparatus 6 is located, it is possible to calculatethe amount of rotation of the terminal apparatus 6. This enables the CPU10 to calculate a change in the direction and the attitude of theterminal apparatus 6 using at least one of the data indicating theacceleration, the data indicating the angular velocity, and the dataindicating the magnetism, which are detected by the terminal apparatus6.

Any calculation method may be used to calculate the attitude of theterminal apparatus 6. For example, a calculation method is possibly usedof correcting the attitude of the terminal apparatus 6, which iscalculated on the basis of the angular velocity indicated by the angularvelocity data Db2, using the acceleration indicated by the accelerationdata Db1 and the direction of the magnetism detected by the magneticsensor 602.

Specifically, the CPU 10 first calculates the attitude of the terminalapparatus 6 on the basis of the angular velocity indicated by theangular velocity data Db2. Any method may be used to calculate theattitude of the terminal apparatus 6 from the angular velocity. Forexample, the attitude of the terminal apparatus 6 may be calculatedusing the most recent attitude (the most recently calculated x-axis,y-axis, and z-axis directions) and the current angular velocity (theangular velocity currently acquired in step 42 in the processing loop).The CPU 10 causes the most recent x-axis, y-axis, and z-axis directionsto rotate about the axes along the respective directions at the currentangular velocity for a unit time, to thereby calculate new x-axis,y-axis, and z-axis directions. It should be noted that the most recentx-axis, y-axis, and z-axis directions are represented by the terminalapparatus perspective direction data De2 and the terminal apparatus updirection data De3, and the current angular velocity is represented bythe angular velocity data Db2. Accordingly, the CPU 10 reads theterminal apparatus perspective direction data De2, the terminalapparatus up direction data De3, and the angular velocity data Db2, andcalculates the attitude of the terminal apparatus 6 (new x-axis, y-axis,and z-axis directions). It should be noted that, as described above, theinitial attitude of the terminal apparatus 6 is defined in step 41described above. Thus, when the attitude of the terminal apparatus 6 iscalculated from the angular velocity, the CPU 10 can calculate thecurrent attitude of the terminal apparatus 6 with respect to the initialattitude of the terminal apparatus 6 that has been calculated first.

Next, the CPU 10 corrects the attitude of the terminal apparatus 6 (thex-axis, y-axis, and z-axis directions), calculated on the basis of theangular velocity, using the acceleration indicated by the accelerationdata Db1. Specifically, the CPU 10 calculates the attitude of theterminal apparatus 6 (the x-axis, y-axis, and z-axis directions) on thebasis of the acceleration indicated by the acceleration data Db1. Here,in the state where the terminal apparatus 6 is substantially stationary,the acceleration applied to the terminal apparatus 6 is thegravitational acceleration. Accordingly, in this state, it is possibleto calculate the direction of the gravitational acceleration (thedirection of gravity) using the direction of the acceleration indicatedby the acceleration data Db1. This makes it possible to calculate thefacing direction of the terminal apparatus 6 relative to the directionof gravity (the x-axis, y-axis, and z-axis directions with respect tothe direction of gravity).

When the attitude of the terminal apparatus 6 based on the accelerationis calculated, the CPU 10 corrects the attitude based on the angularvelocity, using the attitude based on the acceleration. Specifically,the CPU 10 makes a correction to approximate at a predetermined rate theattitude of the terminal apparatus 6 (the x-axis, y-axis, and z-axisdirections) calculated on the basis of the angular velocity to theattitude of the terminal apparatus 6 (the x-axis, y-axis, and z-axisdirections) calculated on the basis of the acceleration. Thepredetermined rate may be a fixed value set in advance, or may be set inaccordance with, for example, the acceleration indicated by theacceleration data Db1. Further, the attitude of the terminal apparatus 6calculated on the basis of the acceleration cannot be calculated in thedirection of rotation about the direction of gravity, and therefore, theCPU 10 may not make a correction on the attitude in this rotationdirection. When correcting, on the basis of the direction of magnetismdetected by the magnetic sensor 602, the attitude of the terminalapparatus 6 calculated on the basis of the angular velocity, the CPU 10may approximate at a predetermined rate the attitude of the terminalapparatus 6 calculated on the basis of the angular velocity to theattitude of the terminal apparatus 6 calculated on the basis of thedirection of magnetism detected by the magnetic sensor 602. This enablesthe CPU 10 to accurately calculate the attitude of the terminalapparatus 6.

Next, the CPU 10 sets the player object Po (step 83), and proceeds tothe subsequent step. With reference to FIG. 18, a description is givenbelow of the player object setting process in step 83 described above.

Referring to FIG. 18, the CPU 10 calculates the difference in horizontalangle between the real space reference direction and the terminalapparatus perspective direction (step 120), and proceeds to thesubsequent step. Here, the difference in horizontal angle describedabove is the difference in angle (e.g., the angle C shown in FIG. 20)obtained by projecting onto a horizontal plane the difference in anglebetween the real space reference direction and the terminal apparatusperspective direction in real space, and indicates the angle by whichthe direction of the terminal apparatus 6 has changed from the initialattitude of the terminal apparatus 6 with respect to the verticaldirection in real space (the direction in which the back surface of theterminal apparatus 6 is directed (the z-axis positive direction shown inFIG. 3)). For example, the CPU 10 calculates the difference inhorizontal angle described above using the real space referencedirection indicated by the real space reference direction data De1 andthe terminal apparatus perspective direction indicated by the terminalapparatus perspective direction data De2.

Next, the CPU 10 calculates the operation indication direction relativeto the virtual world reference direction, in accordance with thedifference in horizontal angle and the attitude (up-down angle) of theterminal apparatus 6 in the up-down direction (step 121), and proceedsto the subsequent step. For example, as shown in FIG. 20, the CPU 10calculates the operation indication direction in the virtual world,using the virtual world reference direction indicated by the virtualworld reference direction data Df1, such that the difference in angleobtained by projecting the virtual world reference direction and theoperation indication direction onto a horizontal plane in the virtualworld is the difference in horizontal angle described above, and thevirtual world reference direction and the operation indication directionhave the same positional relationship (i.e., the positionalrelationships are such that when the terminal apparatus perspectivedirection has rotated to the left relative to the real space referencedirection, also the operation indication direction rotates to the leftrelative to the virtual world reference direction) (the angle C shown inFIG. 20). Further, as shown in FIG. 14, the CPU 10 calculates theoperation indication direction in the virtual world, using the terminalapparatus perspective direction indicated by the terminal apparatusperspective direction data De2, such that the difference in anglebetween the horizontal direction in the virtual world and the operationindication direction is the difference in angle between the horizontaldirection in real space and the terminal apparatus perspectivedirection, and the horizontal direction in the virtual world and theoperation indication direction have the same positional relationship(i.e., when the terminal apparatus perspective direction is directeddownward relative to the horizontal direction in real space, also theoperation indication direction is directed downward relative to thehorizontal direction in the virtual world) (the angle B shown in FIG.14). Then, the CPU 10 updates the operation indication direction dataDf2 using the calculated operation indication direction.

Next, the CPU 10 calculates the operation up direction in the virtualworld on the basis of the terminal apparatus up direction of theterminal apparatus 6 (step 122), and proceeds to the subsequent step.For example, the CPU 10 calculates the operation up direction using theterminal apparatus perspective direction indicated by the terminalapparatus perspective direction data De2, the terminal apparatus updirection indicated by the terminal apparatus up direction data De3, andthe operation indication direction indicated by the operation indicationdirection data Df2. Specifically, as shown in FIG. 14, the CPU 10calculates the operation up direction such that the operation updirection in the virtual world with respect to the operation indicationdirection is the same as the terminal apparatus up direction in realspace with respect to the terminal apparatus perspective direction.Here, the terminal apparatus perspective direction and the terminalapparatus up direction are set as directions in real space with respectto the direction of gravity (the vertical direction). Meanwhile, theoperation indication direction is set as a direction in the virtualworld, and therefore can be handled also as a direction with respect tothe direction of gravity set in the virtual world. Thus, the use of theterminal apparatus perspective direction, the terminal apparatus updirection, and the operation indication direction makes it possible tocalculate the operation up direction in the virtual world correspondingto the terminal apparatus up direction in real space. Then, the CPU 10updates the operation up direction data Df3 using the calculatedoperation up direction.

Next, the CPU 10 determines whether or not the player object Po ismoving (step 123). For example, when a movement vector is set in themovement vector data Dh, the CPU 10 determines that the player object Pois moving. When the player object Po is moving, the CPU 10 proceeds tothe subsequent step 124. On the other hand, when the player object Po isnot moving, the CPU 10 proceeds to the subsequent step 130.

In step 124, the CPU 10 calculates the angle of rotational movement inaccordance with the difference in horizontal angle calculated in step120 described above, and proceeds to the subsequent step. For example,the CPU 10 calculates an angle of rotational movement D (the angle Dshown in FIG. 20) by

D=C×c1

(where C is the difference in horizontal angle, and c1 is a positiveconstant less than 1 (e.g., 0.001))

Next, the CPU 10 changes the virtual world reference direction inaccordance with the angle of rotational movement calculated in step 124described above (step 125), and proceeds to the subsequent step. Forexample, the CPU 10, when the virtual world reference directionindicated by the virtual world reference direction data Df1 is projectedonto a horizontal plane in the virtual world, changes the virtual worldreference direction so as to rotate about the direction of gravity inthe virtual world by the angle of rotational movement, and toapproximate the operation indication direction, to thereby update thevirtual world reference direction data Df1 using the virtual worldreference direction after the change.

Next, on the basis of the operation indication direction indicated bythe operation indication direction data Df2 and the operation updirection indicated by the operation up direction data Df3, the CPU 10sets the facing direction and the attitude of the player object Po inthe virtual world (step 126), and proceeds to the subsequent step. As anexample, the CPU 10 uses the operation indication direction, indicatedby the operation indication direction data Df2, as it is as the facingdirection of the player object Po (the direction in which the playerobject Po is to move). Then, the CPU 10 sets the attitude of the playerobject Po in which it is to roll, such that the operation up directionindicated by the operation up direction data Df3 is the up direction ofthe player object Po in the virtual world (the direction of the backsurface of the torso shown in FIG. 14 in the exemplary game describedabove). Then, the CPU 10 updates the action/attitude data Dg on thebasis of the set facing direction and attitude of the player object Po.Thus, the facing direction and the attitude of the player object Po inthe virtual world are set on the basis of the attitude of the terminalapparatus 6 in real space.

For example, when the user has directed the back surface of the terminalapparatus 6 upward (i.e., has pitched the terminal apparatus 6 upward),the facing direction of the player object Po is set to the elevationdirection at an angle based on the elevation angle of the back surfacewith respect to the vertical direction. Further, when the user hasdirected the back surface of the terminal apparatus 6 downward (i.e.,has pitched the terminal apparatus 6 downward), the facing direction ofthe player object Po is set to the depression direction at an anglebased on the depression angle of the back surface with respect to thevertical direction. Further, when the user has rotated (i.e., rolled)the terminal apparatus 6 about the perspective direction, the playerobject Po rolls in the virtual world in the direction in which theterminal apparatus 6 has rolled and in accordance with the angle bywhich the terminal apparatus 6 has rolled. Further, the left-rightfacing direction of the player object Po is set on the basis of a changein the left-right direction of the terminal apparatus 6 relative to thereal space reference direction. For example, when the terminal apparatus6 has rotationally moved in the yaw direction (see FIG. 3), which is thedirection of rotation about the y-axis, also the facing direction of theplayer object Po changes in accordance with the amount of the rotationalmovement. Specifically, when the terminal apparatus 6 has rotationallymoved about the y-axis to the left as viewed in the y-axis negativedirection, also the facing direction of the player object Po changes tothe left of the player object Po in accordance with the amount of therotational movement. Further, when the user maintains the state of theterminal apparatus 6 being rotationally moved to the left or rightrelative to the real space reference direction, the virtual worldreference direction continuously changes to the left or right. Thus,step 121 described above is performed, whereby, in accordance with thechange in the virtual world reference direction, also the operationindication direction continuously changes to the left or right. Thus,when the user maintains the state where the terminal apparatus 6 isrotationally moved to the left or right relative to the real spacereference direction, also the facing direction of the player object Pocontinuously changes further to the left or right. That is, even ifwishing to greatly change the facing direction of the player object Posuch as changing it in the opposite direction, the user does not need tochange the direction of the terminal apparatus 6 greatly in, forexample, the opposite direction. This makes it possible to improve theoperability of the user.

Next, the CPU 10 performs a player object action setting process (step127), and proceeds to the subsequent step. With reference to FIG. 19, adescription is given below of the player object action setting processperformed in step 127 described above.

Referring to FIG. 19, the CPU 10 determines whether or not the actionmode flag indicated by the movement mode flag data Dm is set to off(step 140). Then, when the action mode flag is set to off, the CPU 10proceeds to the subsequent step 141. On the other hand, when the actionmode flag is set to on, the CPU 10 proceeds to the subsequent step 148.

In step 141, the CPU 10 determines whether or not the total load valueis equal to or greater than a predetermined value. For example, the CPU10 determines whether or not the latest total load value indicated bythe load value data Dc is equal to or greater than the predeterminedvalue. The predetermined value is a threshold for determining that theuser has taken action so as to perform leg bending and extending on theboard-type controller 9. As an example, the predetermined value is setto a value obtained by adding a predetermined proportion (e.g., 20%) tothe weight of the user mounted on the board-type controller 9. Here, theweight of the user mounted on the board-type controller 9 may becalculated on the basis of the total load value (e.g., an average valueof the total load values obtained within a predetermined period), or maybe a value input in advance by the user. Then, when the total load valueis equal to or greater than the predetermined value, the CPU 10 proceedsto the subsequent step 142. On the other hand, when the total load valueis less than the predetermined value, the CPU 10 proceeds to thesubsequent step 144.

In step 142, the CPU 10 sets the object action to the action ofdolphin-kicking, and proceeds to the subsequent step. For example, whenthe total load value is equal to or greater than the predeterminedvalue, the CPU 10 determines that the user has taken a leg bending andextending action on the board-type controller 9. In this case, the CPU10 sets the action of the player object Po to the action of swimming bya dolphin kick style, and also sets the action of the player object Poto the action of dolphin-kicking, to thereby update the action/attitudedata Dg using the set action.

Next, the CPU 10 sets the action mode flag to on (step 143), and endsthe process of this subroutine. For example, the CPU 10 changes theaction mode flag indicated by the movement mode flag data Dm to on.

On the other hand, when it has been determined in step 141 describedabove that the total load value is less than the predetermined value,the CPU 10 determines whether or not the position of the center ofgravity indicated by the position-of-center-of-gravity data Dd has movedinto a right foot area from outside thereof (step 144). When theposition of the center of gravity has moved into the right foot areafrom outside thereof, the CPU 10 proceeds to the subsequent step 145. Onthe other hand, when the position of the center of gravity has not movedinto the right foot area from outside thereof, the CPU 10 proceeds tothe subsequent step 146.

Here, in a predetermined coordinate system corresponding to positions onthe platform 9 a of the board-type controller 9 (e.g., an X1-Y1coordinate system where: the origin is the center of the platform 9 a;an X1-axis direction is the long side direction of the platform 9 a; anda Y1-axis direction is the short side direction), the following are set:an area (right foot area) where the position of the center of gravitymoves when the user has raised their left foot and lowered their rightfoot in a step action; and an area (left foot area) where the positionof the center of gravity moves when the user has raised their right footand lowered their left foot in a step action. For example, the left footarea and the right foot area are set in a left portion and a rightportion, respectively, on the platform 9 a (e.g., a left portion and aright portion when the power button 9 c is placed below), and a neutralarea is set between the left foot area and the right foot area so as toextend in the front-back direction and have a predetermined width. Inthe process of step 144 described above, a positive determination ismade when the latest position of the center of gravity indicated by theposition-of-center-of-gravity data Dd has moved into the right foot areafrom outside thereof, such as when the latest position of the center ofgravity has moved into the right foot area from the neutral area.

It should be noted that the user may possibly step while changing theirdirection on the board-type controller 9. In this case, if the positionsof the left foot area, the right foot area, and the neutral area arefixed, it may not be possible to accurately make step determinations. Insuch a case, the left foot area, the right foot area, and the neutralarea in the coordinate system may be moved on the basis of the movementof the position of the center of gravity in the coordinate system. Forexample, when the moving distance of the position of the center ofgravity during a predetermined period is equal to or greater than apredetermined length, the neutral area is moved so as to extendperpendicular to the movement path of the position of the center ofgravity during the predetermined period in the central position of themovement path, and the left foot area and the right foot area are movedin accordance with the movement of the neutral area. The left foot area,the right foot area, and the neutral area may be thus moved, whereby itis possible to accurately make step determinations even if the user hasstepped while changing their direction on the board-type controller 9.

It should be noted that in the step determination described above, it isdetermined whether the foot with which the user has stepped on theboard-type controller 9 is their left foot or right foot. It may not benecessary to distinguish between left and right. For example, in theabove descriptions, if processing is performed such that the “left foot”is “one foot”, and the “right foot” is the “other foot”, the foot withwhich the user has stepped and the leg to be kicked by the player objectPo with a flutter kick may be the opposite to those of the processingdescribed above. It is, however, possible to instruct an action so as tosynchronize the step action of the user with the timing of the flutterkick action of the player object Po.

In step 145, the CPU 10 sets the object action to the action offlutter-kicking with its right foot, and ends the process of thesubroutine. For example, when the position of the center of gravity hasmoved into the right foot area from outside thereof, the CPU 10determines that the user has taken a step action so as to raise theirleft foot and lower their right foot on the board-type controller 9. Inthis case, the CPU 10 sets the action of the player object Po to theaction of swimming by a flutter kick style, and also sets the action ofthe player object Po to the action of flutter-kicking with its rightfoot, to thereby update the action/attitude data Dg using the setaction.

In step 146, the CPU 10 determines whether or not the position of thecenter of gravity indicated by the position-of-center-of-gravity data Ddhas moved into the left foot area from outside thereof. When theposition of the center of gravity has moved into the left foot area fromoutside thereof, the CPU 10 proceeds to the subsequent step 147. On theother hand, when the position of the center of gravity has not movedinto the left foot area outside thereof, the CPU 10 proceeds to thesubsequent step 151.

In step 147, the CPU 10 sets the object action to the action offlutter-kicking with its left foot, and ends the process of thesubroutine. For example, when the position of the center of gravity hasmoved into the left foot area from outside thereof, the CPU 10determines that the user has taken a step action so as to raise theirright foot and lower their left foot on the board-type controller 9. Inthis case, the CPU 10 sets the action of the player object Po to theaction of swimming by a flutter kick style, and also sets the action ofthe player object Po to the action of flutter-kicking with its leftfoot, to thereby update the action/attitude data Dg using the setaction.

When it has been determined in step 140 described above that the actionmode flag is on, the CPU 10 determines whether or not a predeterminedperiod has elapsed since the action mode flag has been set to on (step148). When the predetermined period has elapsed, the CPU 10 proceeds tothe subsequent step 149. On the other hand, when the predeterminedperiod has not elapsed, the CPU 10 proceeds to the subsequent step 150.Here, the predetermined period is a period from the time when a dolphinkick is performed to the time when the next dolphin kick operation inputor the next flutter kick operation input is permitted. Thus, thedetermination of the action of the user using the board-type controller9 is not made until the predetermined period elapses. This makes itpossible to prevent an erroneous determination of the action of the userimmediately after dolphin-kicking.

In step 149, the CPU 10 sets the action mode flag to off, and ends theprocess of the subroutine. For example, the CPU 10 changes the actionmode flag indicated by the movement mode flag data Dm to off.

On the other hand, in step 150, the CPU 10 sets the object action to“during a dolphin kick”, and ends the process of the subroutine. Forexample, the CPU 10 continuously sets the action of the player object Poto the action of swimming by a dolphin kick style, and also sets theaction of the player object Po to “during a dolphin kick”, to therebyupdate the action/attitude data Dg using the set action.

When the action mode flag is off (a positive determination is made instep 140 described above); the total load value is less than thepredetermined value (a negative determination is made in step 141described above); the position of the center of gravity has not movedinto the right foot area from outside thereof (a negative determinationis made in step 144 described above); and the position of the center ofgravity has not moved into the left foot area from outside thereof (anegative determination is made in step 146 described above), the CPU 10sets the currently set action to a continuous action (step 151), andends the process of the subroutine. For example, when the action ofkicking legs (e.g., the action of flutter-kicking) is set, the CPU 10cancels the action, and continuously sets the swimming style of theplayer object Po that is set as the object action, to thereby update theaction/attitude data Dg using the set action.

Referring back to FIG. 18, after the player object action settingprocess in step 127 described above, the CPU 10 sets the movement vectorof the player object Po (step 128), and proceeds to the subsequent step.For example, the CPU 10 calculates the moving velocity of the playerobject Po on the basis of the chronological array of the total loadvalues indicated by the load value data Dc (the history of changes inthe total load value) and the action (swimming style) of the playerobject Po set in step 127 described above. As an example, the CPU 10makes a calculation such that the shorter the cycle of the total loadvalue changing, the greater the velocity of the player object Po moving.As another example, the CPU 10 makes a calculation such that the greaterthe maximum value of, or the width of the change in, the total loadvalue, the greater the velocity of the player object Po moving. Forexample, the shorter the interval of the user stepping on the board-typecontroller 9, the greater the moving velocity of the player object Poswimming with a flutter kick. Further, the shorter the interval of theuser performing leg bending and extending on the board-type controller9, the greater the moving velocity of the player object Po swimming witha dolphin kick. The greater the strength of leg bending and extending,the greater the moving velocity of the player object Po swimming with adolphin kick. Then, the CPU 10 corrects the calculated moving velocitywith a coefficient based on the set swimming style, and determines themoving velocity of the player object Po. Further, the moving velocity ofthe player object Po is set so as to be relatively great at the time ofkicking legs, and set so as to decrementally decrease until the nexttime of kicking legs.

In addition, the CPU 10 sets the operation indication direction in thevirtual world, indicated by the operation indication direction data Df2,to the moving direction of the player object Po. Then, the CPU 10calculates the movement vector of the player object Po in the virtualworld using the moving velocity and the moving direction of the playerobject Po that have been calculated as described above, to therebyupdate the movement vector data Dh using the calculated movement vector.For example, the CPU 10 sets the length of the movement vector using thecalculated moving velocity of the player object Po, and sets thedirection of the movement vector using the operation indicationdirection indicated by the operation indication direction data Df2.

Next, the CPU 10 calculates the position of the player object Po (step129), and ends the process of this subroutine. For example, on the basisof the movement vector indicated by the movement vector data Dh, the CPU10 moves, in the virtual world, the position of the player object Poindicated by the position data Di, to thereby update the position dataDi using the position after the movement.

On the other hand, when it has been determined in step 123 describedabove that the player object Po is not moving, the CPU 10 sets thefacing direction of the player object Po in the virtual world on thebasis of the operation indication direction indicated by the operationindication direction data Df2 and the operation up direction data Df3(step 130), and proceeds to the subsequent step. It should be noted thatthe process of step 130 described above is similar to the process ofstep 126 described above, and therefore is not described in detail here.

Next, the CPU 10 sets the object action to a static state (step 131),and ends the process of the subroutine. For example, the CPU 10 sets theaction of the player object Po to a static state (e.g., the state ofdrifting without swimming underwater), to thereby update theaction/attitude data Dg using the set action.

Referring back to FIG. 17, after the player object setting process instep 83 described above, the CPU 10 sets parameters concerning thesecond virtual camera (step 84), and proceeds to the subsequent step.For example, a terminal game image and a monitor game image aregenerated as, for example, three-dimensional CG images obtained bycalculating a game space viewed from a virtual camera placed in thevirtual world (virtual space). Specifically, the first virtual camerafor generating a terminal game image is set so as to include the stateof the virtual world viewed from behind and close to the player objectPo placed in the virtual world. Further, the second virtual camera forgenerating a monitor game image is set in the same virtual world wherethe first virtual camera is set, the second virtual camera set so as toinclude the state of the virtual world obtained by viewing from adistant bird's-eye view the player object Po placed in the virtualworld. The CPU 10 sets the positions of the first virtual camera and thesecond virtual camera in the virtual world (parameters concerning thefirst virtual camera and the second virtual camera) on the basis of theposition of the player object Po indicated by the position data Di andthe operation indication direction based on the operation indicationdirection data Df2 (the direction of the player object Po). A terminalgame image and a monitor game image are game images of the virtual worldthat are thus viewed from different points of view. This causes the gameimages of the virtual world viewed from the different points of view tobe displayed on the LCD 61 and the monitor 2.

Specifically, the CPU 10 controls the attitude of the first virtualcamera in the virtual world so as to be the same as the attitude of theterminal apparatus 6 in real space. That is, the CPU 10 controls theattitude of the first virtual camera such that: the operation indicationdirection set on the basis of the z-axis positive direction of theterminal apparatus 6 (the operation indication direction indicated bythe operation indication direction data Df2) corresponds to the Z-axispositive direction (the direction of the line of sight) of the firstvirtual camera; and the operation up direction set on the basis of they-axis positive direction of the terminal apparatus 6 (the operation updirection indicated by the operation up direction data Df3) correspondsto the Y-axis positive direction of the first virtual camera (thevirtual camera up direction). Then, the positional relationship(including the roll directions) between the player object Po and thefirst virtual camera is fixed, and the position of the first virtualcamera is set on the basis of the position of the player object Po thatis moving in the virtual world. The first virtual camera is thus set forthe player object Po in the virtual world, whereby images of the virtualworld as shown in FIGS. 11A, 12A, and 13A can be displayed on the LCD 61of the terminal apparatus 6 having attitudes as shown in FIGS. 11B, 12B,and 13B. That is, the user can control the actions of the player objectPo and the first virtual camera for generating a virtual world image tobe displayed on the LCD 61 of the terminal apparatus 6, by moving theterminal apparatus 6. Then, the attitude and the direction of theterminal apparatus 6 are in conjunction with the attitude and thedirection of the first virtual camera. This enables the user to enjoy afeeling as if peeping at the virtual world using the LCD 61 of theterminal apparatus 6 as a peep window.

Next, the CPU 10 sets the movement path of the player object Po (step85), and ends the process of this subroutine. For example, the CPU 10adds the current position of the player object Po, indicated by theposition data Di, to the movement path indicated by the movement pathdata Dk, to thereby update the movement path data Dk using the movementpath to which the current position has been added.

Referring back to FIG. 16, after the game control process in step 44,the CPU 10 and the GPU 32 generate a monitor game image to be displayedon the monitor 2 (step 45), and proceed to the subsequent step. Forexample, the CPU 10 and the GPU 32 read from the main memory the dataindicating the result of the game control process performed in step 44,and read from the VRAM 34 the data used to generate a monitor gameimage. Then, the CPU 10 and the GPU 32 generate a game image using theread data, and store the generated monitor game image in the VRAM 34.Any monitor game image may be generated by any method so long as themonitor game image represents the result of the game control processperformed in step 44. For example, the monitor game image may be athree-dimensional CG image generated by the steps of: placing the secondvirtual camera in the virtual world on the basis of the parametersconcerning the second virtual camera that are indicated by the virtualcamera data Dj; placing the player object Po in the virtual world on thebasis of the action/attitude data Dg and the position data Di; placingthe movement path Lp in the virtual world on the basis of the movementpath data Dk; and calculating the virtual world viewed from the secondvirtual camera.

Next, the CPU 10 and the GPU 32 generate a terminal game image to bedisplayed on the terminal apparatus 6 (step 46), and proceed to thesubsequent step. For example, the CPU 10 and the GPU 32 read from themain memory the data indicating the result of the game control processperformed in step 44, and read from the VRAM 34 the data used togenerate a terminal game image. Then, the CPU 10 and the GPU 32 generatea terminal game image using the read data, and store the generatedterminal game image in the VRAM 34. Similarly to the monitor game image,any terminal game image may be generated by any method so long as theterminal game image represents the result of the game control processperformed in step 44. Further, the terminal game image may be generatedby the same method as, or a different method from, that for the monitorgame image. For example, the terminal game image may be generated as athree-dimensional CG image by the steps of: placing the first virtualcamera in the virtual world on the basis of the parameters concerningthe first virtual camera that are indicated by the virtual camera dataDj; placing the player object Po in the virtual world on the basis ofthe action/attitude data Dg and the position data Di; and calculatingthe virtual world viewed from the first virtual camera.

It should be noted that if the action of the player object Po indicatedby the action/attitude data Dg indicates the action of kicking legs (theaction of dolphin-kicking or the action of flutter-kicking), the playerobject Po may be placed in the virtual world to as to take the action ofkicking legs. This makes it possible to cause the player object Po totake action in synchronization with the action (a step action or a legbending and extending action) taken by the user on the board-typecontroller 9.

Next, the CPU 10 generates a monitor game sound to be output to theloudspeakers 2 a of the monitor 2 (step 47), and proceeds to thesubsequent step. For example, the CPU 10 causes the DSP 33 to generate amonitor game sound to be output from the loudspeakers 2 a, in accordancewith the result of the game control process performed in step 44. As anexample, the CPU 10 causes the DSP 33 to generate a monitor game soundin which BGM or the like to be output from the monitor 2 is added to thevoices and the action sounds of the objects, sound effects, and the likethat are supposed to be heard on the basis of the position of the secondvirtual camera in the virtual world set in accordance with the result ofthe game control process in step 44.

Next, the CPU 10 generates a terminal game sound to be output to theloudspeakers 607 of the terminal apparatus 6 (step 48), and proceeds tothe subsequent step. For example, the CPU 10 causes the DSP 33 togenerate a terminal game sound to be output from the loudspeakers 607,in accordance with the result of the game control process performed instep 44. As an example, the CPU 10 causes the DSP 33 to generate aterminal game sound in which BGM or the like to be output from theterminal apparatus 6 is added to the voices and the action sounds of theobjects, sound effects, and the like that are supposed to be heard onthe basis of the position of the first virtual camera in the virtualworld set in accordance with the result of the game control process instep 44. The terminal game sound may be the same as, or different from,the monitor game sound. Alternatively, the terminal game sound may bepartially different from the monitor game sound (e.g., the terminal gamesound and the monitor game sound include the same BGM and differentsound effects). It should be noted that when the monitor game sound andthe terminal game sound are the same, the terminal game sound generationstep in step 48 may not need to be performed.

Next, the CPU 10 generates a terminal game sound to be output to theloudspeakers 607 of the terminal apparatus 6 (step 49), and proceeds tothe subsequent step. For example, the CPU 10 causes the DSP 33 togenerate a terminal game sound to be output from the loudspeakers 607,in accordance with the result of the game control process performed instep 44. As an example, the CPU 10 causes the DSP 33 to generate aterminal game sound in which BGM or the like to be output from theterminal apparatus 6 is added to the voices and the action sounds of theobjects, sound effects, and the like that are supposed to be heard onthe basis of the position of the first virtual camera in the virtualworld set in accordance with the result of the game control process instep 44. The terminal game sound may be the same as, or different from,the monitor game sound. Alternatively, the terminal game sound may bepartially different from the monitor game sound (e.g., the terminal gamesound and the monitor game sound include the same BGM and differentsound effects). It should be noted that when the monitor game sound andthe terminal game sound are the same, the terminal game sound generationstep in step 48 may not need to be performed.

Next, the CPU 10 transmits the terminal game image and the terminal gamesound to the terminal apparatus 6 (step 50), and proceeds to thesubsequent step. For example, the CPU 10 transmits to the codec LSI 27the data of the terminal game image stored in the VRAM 34 and the dataof the terminal game sound generated by the DSP 33. The codec LSI 27performs a predetermined compression process on the transmitted data.The compressed data of the terminal game image and the compressed dataof the terminal game sound are transmitted from the codec LSI 27 to theterminal communication module 28, and then transmitted from the terminalcommunication module 28 to the terminal apparatus 6 via the antenna 29.The data of the terminal game image and the data of the terminal gamesound that have been transmitted from the game apparatus body 5 arereceived by the wireless module 610 of the terminal apparatus 6, and aresubjected to a predetermined decompression process by the codec LSI 606.Then, the decompressed data of the terminal game image is output to theLCD 61, and the decompressed data of the terminal game sound is outputto the sound IC 608. This causes the terminal game image to be displayedon the LCD 61, and causes the terminal game sound to be output from theloudspeakers 607.

Next, the CPU 10 determines whether or not the game is to be ended (step51). Conditions for ending the game may be, for example: that particularconditions have been satisfied so that the game is over, or the game iscompleted; or that the user has performed an operation for ending thegame. When the game is not to be ended, the CPU 10 returns to step 42and repeats the same processing. On the other hand, when the game is tobe ended, the CPU 10 ends the processing of the flow chart. Thereafter,the series of processes 42 through 51 is repeatedly performed until theCPU 10 determines in step 51 that the game is to be ended.

As described above, the processing described above makes it possiblethat when an image of the virtual world is displayed on the terminalapparatus 6 capable of allowing the user to view a screen thereof whileholding it, a suitable image is displayed on the terminal apparatus 6,regardless of the direction in which the user is holding the terminalapparatus 6. Further, the processing described above enables the user tocontrol the actions (attitudes) of the player object Po and the firstvirtual camera for generating a virtual world image to be displayed onthe LCD 61 of the terminal apparatus 6, by moving the terminal apparatus6. Then, the attitude and the direction of the terminal apparatus 6 arein conjunction with the attitude and the direction of the first virtualcamera. This enables the user to enjoy a feeling as if peeping at thevirtual world using the LCD 61 of the terminal apparatus 6 as a peepwindow.

It should be noted that in the exemplary game described above, thevirtual camera (first virtual camera) for generating an image to bedisplayed on the LCD 61 is controlled (the position, the direction, andthe attitude of the virtual camera are controlled) on the basis of theattitude of the terminal apparatus 6. Such control makes it possible toprovide the user with an image as if peeping at the virtual worldthrough the LCD 61, and provide the user with a feeling as if being inthe virtual world. Further, the operation using the attitude of theterminal apparatus 6 enables the operation of rotating the terminalapparatus 6 in three directions, such as a left-right swing (yaw) aboutthe vertical direction (e.g., about the y-axis direction), an upward anddownward swing (pitch) about the left-right horizontal direction (e.g.,about the x-axis direction), and a left-right rotation (roll) about thefront-back horizontal direction (e.g., about the z-axis direction), andtherefore is suitable for controlling the virtual camera capable ofmaking a similar movement also in the virtual world. Thus, the attitudeof the virtual camera in the virtual world may be controlled so as tocoincide with the attitude of the terminal apparatus 6 in real space,whereby it is possible to provide an image as if peeping in thedirection desired by the user in the virtual world. In addition, in theexemplary game described above, in accordance with the user takingaction on the board-type controller 9, the player object takes action(e.g., a moving action). That is, the user is provided, by an imagedisplayed on the LCD 61, with a feeling as if being in the virtualworld, and is additionally provided with an operation feeling as if theuser themselves is a player object in real space. This enhances thefeeling as if being in the virtual world.

In addition, in the exemplary game described above, the attitude of theplayer object Po displayed on the LCD 61 is controlled on the basis ofthe attitude of the terminal apparatus 6. Such control makes it possibleto provide the user with an operation environment as if the terminalapparatus 6 were the player object Po, and also provide a feeling as ifthe user were the player object Po in the virtual world, and a feelingas if directly controlling the player object Po. Further, the operationusing the attitude of the terminal apparatus 6 enables the operation ofrotating the terminal apparatus 6 in three directions, such as aleft-right swing (yaw) about the vertical direction (e.g., about they-axis direction), an upward and downward swing (pitch) about theleft-right horizontal direction (e.g., about the x-axis direction), anda left-right rotation (roll) about the front-back horizontal direction(e.g., about the z-axis direction), and therefore is suitable forcontrolling the player object Po capable of making a similar movementalso in the virtual world. For example, in the exemplary game, aleft-right swing (yaw) about the height direction along the LCD 61 ofthe terminal apparatus 6 (the y-axis direction) may be set to correspondto a change in the attitude of the player object Po in the left-rightdirection (yaw); an upward and downward swing (pitch) about theleft-right direction along the LCD 61 (the x-axis direction) may be setto correspond to a change in the attitude of the player object Po in theup-down direction (pitch); and a left-right rotation (roll) about theperspective direction of the LCD 61 (the z-axis direction) may be set tocorrespond to a rotational change in the attitude of the player objectPo about the forward direction (moving direction) of the player objectPo (roll), whereby it is possible to provide an image of an object thatchanges its attitude to the attitude desired by the user in the virtualworld. Further, in the exemplary game described above, in accordancewith the user taking action on the board-type controller 9, the playerobject Po takes action (e.g., a moving action). That is, the usercontrols one player object Po using a plurality of devices (the terminalapparatus 6 and the board-type controller 9). This makes it possible toperform an unprecedented operation. Further, the control of the attitudeof the player object Po on the basis of the attitude of the terminalapparatus 6 makes it possible to provide a feeling as if the user werethe player object Po in the virtual world, and a feeling as if directlycontrolling the player object Po. As well as this, the operationperformed on the board-type controller 9 makes it possible to provide anoperation feeling as if the user themselves were the player object Po inreal space. This enhances the feeling as if being in the virtual world.Further, in the exemplary game described above, it is possible to causethe player object Po to move in the forward direction of the playerobject Po. Thus, on the basis of the attitude of the terminal apparatus6, the user can set the attitude (forward direction) of the playerobject Po, and can also set the moving direction of the player objectPo. This enables the operation of setting the moving direction in anintuitive manner, which facilitates the setting of the moving directionto the direction desired by the user. Further, similarly to the attitudeof the player object Po described above, it is possible to set themoving direction of the player object Po in accordance with theoperation of rotating the terminal apparatus 6 in three directions. Thismakes it possible to set in the virtual world the moving directiondesired by the user as it is.

In addition, in the exemplary game described above, the moving directionof the player object Po is set on the basis of the attitude of theterminal apparatus 6. Such a setting of the moving direction makes itpossible to provide the user with an operation environment as if theterminal apparatus 6 were the player object Po, and also provide afeeling as if the user were the player object Po in the virtual world,and a feeling as if directly controlling the player object Po. Further,the operation using the attitude of the terminal apparatus 6 enables theoperation of rotating the terminal apparatus 6 in three directions, suchas a left-right swing (yaw) about the vertical direction (e.g., aboutthe y-axis direction), an upward and downward swing (pitch) about theleft-right horizontal direction (e.g., about the x-axis direction), anda left-right rotation (roll) about the front-back horizontal direction(e.g., about the z-axis direction), and therefore is suitable forcontrolling the player object Po whose moving direction can be set in atleast one of the three directions also in the virtual world. Forexample, in the exemplary game, a left-right swing (yaw) about theheight direction along the LCD 61 of the terminal apparatus 6 (they-axis direction) may be set to correspond to a change in the movingdirection of the player object Po in the left-right direction (yaw), andan upward and downward swing (pitch) about the left-right directionalong the LCD 61 (the x-axis direction) may be set to correspond to achange in the moving direction of the player object Po in the up-downdirection (pitch), whereby it is possible to provide the control of anobject in which the object moves in the virtual world in the movingdirection desired by the user. Further, in the exemplary game describedabove, the moving action of the player object Po (e.g., the setting ofthe moving velocity) is taken in accordance with the user taking actionon the board-type controller 9. That is, the user controls the movingdirection and the moving velocity of one player object Po using aplurality of devices (the terminal apparatus 6 and the board-typecontroller 9). This enables an unprecedented operation. Further, thesetting of the moving direction of the player object Po based on theattitude of the terminal apparatus 6 provides a feeling as if the userwere the player object Po in the virtual world, and a feeling as ifdirectly controlling the player object Po, and also provides, on thebasis of the operation performed on the board-type controller 9, anoperation feeling as if the user themselves were the player object Po inreal space. This enhances the feeling as if being in the virtual world.

In addition, in the exemplary game described above, it is possible toset the perspective direction in the virtual world displayed on the LCD61 of the terminal apparatus 6, as the moving direction of the playerobject Po. This enables the user to set the moving direction of theplayer object Po on the basis of the attitude of the terminal apparatus6. Further, the virtual world is displayed on the LCD 61 such that themoving direction is the perspective direction. This enables theoperation of setting the moving direction in an intuitive manner, whichfacilitates the setting of the moving direction to the direction desiredby the user. Further, similarly to the direction of the virtual cameradescribed above, it is possible to set the moving direction of theplayer object Po in accordance with the operation of rotating theterminal apparatus 6 in three directions. This makes it possible to setin the virtual world the moving direction desired by the user as it is.

In addition, in the exemplary game described above, an image of thevirtual world including at least the player object Po is displayed onthe LCD 61 of the terminal apparatus 6. Alternatively, an image of thevirtual world in another form may be displayed on the LCD 61. Forexample, a virtual world image viewed from the first-person point ofview of the player object Po may be displayed on the LCD 61 withoutdisplaying the player object Po. In this case, the up-down direction inthe virtual world displayed on the LCD 61 from the first-person point ofview may, in accordance with the roll of the terminal apparatus 6, rollin the opposite direction (i.e., the first virtual camera in the virtualworld may roll in the same action as that of the roll of the terminalapparatus 6 in real space). Further, the perspective direction of thevirtual world displayed on the LCD 61 from the first-person point ofview may be the same as, or may be different from, the moving directionof the player object Po. It is needless to say that when the perspectivedirection of the virtual world displayed on the LCD 61 from thefirst-person point of view is the same as the moving direction of theplayer object Po, the virtual world is displayed on the LCD 61 such thatthe moving direction is the perspective direction, and this makes itpossible to perform the operation of setting the moving direction in anintuitive manner, which facilitates the setting of the moving directionto the direction desired by the user.

It should be noted that in the exemplary game described above, theexemplary processing is performed such that in accordance with theoperation indication direction determined on the basis of the attitudeof the terminal apparatus 6, the position and the attitude of thevirtual camera are controlled in conjunction immediately after thedetermination. In accordance with the change in the operation indicationdirection and the change in the operation up direction, however, theposition and the attitude of the first virtual camera may be controlledafter a delay of a predetermined period. In this case, after the facingdirection and the attitude of the player object Po change, the positionand the attitude of the first virtual camera change so as to follow thefacing direction and the attitude of the player object Po after thedelay of the predetermined period.

In addition, in the exemplary game described above, the positionalrelationship (including the roll directions) between the player objectPo and the first virtual camera is fixed, and the position of the firstvirtual camera is set on the basis of the position of the player objectPo that is moving in the virtual world. Alternatively, the positionalrelationship between the player object Po and the first virtual cameramay change. As an example, if the operation of rotating the terminalapparatus 6 is performed, such as a left-right rotation (roll) about thefront-back horizontal direction (e.g., about the z-axis direction), thefirst virtual camera is caused to roll in the virtual world about thedirection of the line of sight of the first virtual camera (the Z-axispositive direction) in the direction in which the terminal apparatus 6has rolled and by the angle of the roll. In contrast, the player objectPo is caused to roll in the virtual world about the moving direction(the same direction as the direction of the line of sight of the firstvirtual camera if the moving direction coincides with the direction ofthe line of sight) in the direction of the roll and by an angle smallerthan the angle of the roll. This changes at least the positionalrelationship between the player object Po and the first virtual camerain the roll directions. Even when the roll angle by which the attitudeof the player object Po thus changes in the roll direction is smallerthan the roll angle of the terminal apparatus 6, the player object Postill takes the action of rolling in response to the operation of theuser rolling the terminal apparatus 6. This makes it possible to enjoy asimilar operation feeling.

As another example, if the operation of rotating the terminal apparatus6 is performed, such as a left-right rotation (roll) about thefront-back horizontal direction (e.g., about the z-axis direction), thefirst virtual camera is caused to roll in the virtual world about thedirection of the line of sight of the first virtual camera (the Z-axispositive direction) in the direction in which the terminal apparatus 6has rolled and by the angle of the roll. In contrast, the player objectPo is once caused to roll in the virtual world about the movingdirection (the same direction as the direction of the line of sight ofthe first virtual camera if the moving direction coincides with thedirection of the line of sight) in the direction of the roll and by thesame angle as the angle of the roll. Then, the attitude of the playerobject Po is changed at a predetermined roll rotation velocity such thatonly the direction of the back surface of the torso of the player objectPo is directed upward in the virtual world. This changes at least thepositional relationship between the player object Po and the firstvirtual camera in the roll directions after the movement of rollrotation. Even when control is thus performed such that the attitude ofthe player object Po in the roll direction returns to a direction basedon the up-down direction in the virtual world, it is possible to causethe player object Po to once take the action of rolling in response tothe operation of the user rolling the terminal apparatus 6. This enablesthe user to enjoy a similar operation feeling when having performed theoperation of rolling the terminal apparatus 6.

In addition, in the above descriptions, the attitude of the firstvirtual camera in the virtual world is controlled so as to be the sameas the attitude of the terminal apparatus 6 in real space. This makes itpossible, in accordance with the user directing the terminal apparatus 6in the direction that they wish to view, to provide the user with, forexample, an image as if peeping at the virtual world through theterminal apparatus 6, and provide the user with a feeling as if being inthe virtual world. If, however, such effects are not expected, theattitude of the terminal apparatus 6 in real space and the attitude ofthe first virtual camera in the virtual world do not need to completelycoincide with each other. For example, when the terminal apparatus 6 hasrotated about a predetermined axis (e.g., the z-axis) by an amount ofrotation θt, the first virtual camera may be caused to rotate about adirection in the virtual world corresponding to the predetermined axis(e.g., the direction of the line of sight of the first virtual camera)by an amount of rotation smaller than, or greater than, the amount ofrotation θt. Further, after the first virtual camera rotates by theamount of rotation smaller than, or greater than, the amount of rotationθt, the first virtual camera may further rotate at a predeterminedrotation velocity so that the attitude of the first virtual camera isthe same as the attitude of the terminal apparatus 6 in real space.

In addition, in the exemplary information processing described above,the terminal apparatus up direction (the y-axis positive direction) andthe operation up direction are used to set the motion of the terminalapparatus 6 about the z-axis and the directions in which the firstvirtual camera and the player object Po are caused to roll.Alternatively, once the attitudes of the first virtual camera and theplayer object Po in the virtual world in the roll directions are set tocorrespond to the attitude of the terminal apparatus 6, these directionsmay not need to be used in the subsequent processing. For example, afterthe attitudes of the first virtual camera and the player object Po inthe virtual world in the roll directions are set to correspond to theattitude of the terminal apparatus 6, the angle of rotationcorresponding to the angular velocity generated about the z-axis of theterminal apparatus 6 may be set as it is to correspond to the attitudesof the first virtual camera and the player object Po in the virtualworld in the roll directions, whereby it is possible to control theattitudes in a similar manner.

In addition, in the above descriptions, the virtual world is fixed, andthe virtual camera and the player object Po are caused to move relativeto the virtual world. It is, however, needless to say that the virtualcamera and the player object Po only need to move relative to thevirtual world. For example, when the direction, the position, theattitude, and the like of the virtual camera in the virtual world changein accordance with changes in the direction, the motion, and theattitude, and the like of the terminal apparatus 6, the direction, theposition, the attitude, and the like of the virtual camera may be fixed,and the entire virtual world may be moved with respect to the virtualcamera in accordance with changes in the direction, the motion, and theattitude, and the like of the terminal apparatus 6. Further, when thedirection, the position, the attitude, and the like of the player objectPo in the virtual world change in accordance with changes in thedirection, the motion, and the attitude, and the like of the terminalapparatus 6, the direction, the position, the attitude, and the like ofthe player object Po may be fixed, and the entire virtual world exceptfor the player object Po may be moved with respect to the player objectPo in accordance with changes in the direction, the motion, and theattitude, and the like of the terminal apparatus 6.

In addition, in the above descriptions, the terminal apparatus updirection (the y-axis positive direction) and the operation up directionare used to set the motion of the terminal apparatus 6 about the z-axisand the directions in which the first virtual camera and the playerobject Po are caused to roll. Alternatively, the above motion anddirections may be set using another axis orthogonal to the z-axis, oranother direction orthogonal to the direction of the line of sight ofthe first virtual camera and the moving direction of the player objectPo (the operation indication direction). For example, the motion of theterminal apparatus 6 about the z-axis and the directions in which thefirst virtual camera and the player object Po are caused to roll may beset using the terminal apparatus left direction (the x-axis positivedirection) and the operation left direction.

In addition, in the exemplary game described above, the game imagedisplayed on the LCD 61 of the terminal apparatus 6 and the game imagedisplayed on the monitor 2 are images both representing the state of thesame virtual world (virtual space), but are images different from eachother in the point of view, and the range of view, toward the virtualworld (virtual space) is viewed. This enables the user to view thevirtual world (virtual space) displayed on the two display screens indifferent fields of view and different display ranges, and thereforeenables the user to appropriately view a suitable game image dependingon the state of the game. Further, the exemplary game described aboveenables the user to perform an operation while holding the terminalapparatus 6, to thereby change the positions and the attitudes of theplayer object Po and the virtual camera in accordance with the attitudeand the position of the terminal apparatus 6 in real space, and alsochange an image displayed on the LCD 61 in accordance with the positionsand the attitudes of the player object Po and the virtual camera. Thismakes it possible to provide a sense of presence in the virtual world(virtual space) to the user viewing an image displayed on the LCD 61while holding the terminal apparatus 6. On the other hand, viewing onlyan image displayed on the LCD 61 may make it difficult to understand theposition relative to the entire virtual world (virtual space) and thecircumstance of the player object Po. The display of the virtual world(virtual space) in a relatively wide range on the monitor 2 can solvesuch a problem.

In addition, in the exemplary game described above, the user can performan operation based on a change in the load applied to the board-typecontroller 9, and can also perform an operation holding the terminalapparatus 6 (an operation based on the attitude and the position of theapparatus body, a touch operation, a button operation, and the like), onwhich a game image including the player object Po is displayed. Then,the player object Po displayed on the terminal apparatus 6 takes actionin the virtual world in accordance with the operation based on a changein the load applied to the board-type controller 9 and/or the operationusing the terminal apparatus 6. This makes it possible to provide theuser with a feeling as if the user, as the player object Po, took actionand viewed the virtual world, and to provide the user with a feeling asif controlling the player object Po in real space.

Here, in the operation using the board-type controller 9 describedabove, the presence or absence of a load to be applied to the board-typecontroller 9, a change in the load, and the position of the center ofgravity of the load are used to control various actions of the playerobject Po. For example, in the exemplary game described above, theaction, the moving velocity, the moving direction, and the like of theplayer object are changed on the basis of a change in the load appliedto the board-type controller 9, and the position of the center ofgravity of the load. It should be noted that in the exemplary game,directions in the virtual world (e.g., the moving direction, the forwarddirection, and the like of the player object Po) are set in accordancewith the operation direction of the user; however, the directions areset on the basis of the attitude (direction) of the terminal apparatus6. This is an example where, when the operation based on the attitude ofthe terminal apparatus 6 is compared to the operation using theboard-type controller 9, the operation based on the attitude of theterminal apparatus 6 is selected because it makes it easier to performthe operation of setting directions, and increases a sense of presencein the game. Alternatively, in accordance with the content of the game,the more appropriate of the attitude of the terminal apparatus 6 and theposition of the center of gravity of the load to be applied to theboard-type controller 9 may be selected for the operation of indicatingdirections. When the terminal apparatus 6 and the board-type controller9 are thus used as operation means, it is possible to select a suitablemethod from among a plurality of optional operations in order to setdirections in the virtual world in accordance with an operation of theuser.

It should be noted that the game system 1 allows the user to performvarious games using the terminal apparatus 6 and the board-typecontroller 9 as operation means. The terminal apparatus 6 can be used asa controller that allows the user to provide an input by an operationbased on the motion of the body of the terminal apparatus 6, a touchoperation, a button operation, or the like, while it can be used as aportable display or a second display. Accordingly, the game system 1achieves a wide range of games. That is, the terminal apparatus 6functions as a display apparatus, and therefore, there may be a gamesystem in which: the terminal apparatus 6 is used as display means whilethe monitor 2 and the controller 7 are not used; and the board-typecontroller 9 is used as operation means. Further, the terminal apparatus6 functions as an operation device as well as a display apparatus, andtherefore, there may be a game system in which the terminal apparatus 6is used as display means while the monitor 2 and the controller 7 arenot used, and the terminal apparatus 6 and the board-type controller 9are used as operation means. Further, the terminal apparatus 6 functionsas an operation device as well as a display apparatus, and therefore,there may be a game system in which the terminal apparatus 6 is used asdisplay means while the monitor 2, the board-type controller 9, and thecontroller 7 are not used, and the terminal apparatus 6 is used asoperation means.

In addition, in the exemplary embodiment, the terminal apparatus 6functions as a so-called thin client terminal, which does not performgame processing. In the exemplary embodiment, however, at least a partof the series of steps in the game processing to be performed by thegame apparatus body 5 may be performed by the terminal apparatus 6. Asan example, the terminal game image generation process may be performedby the terminal apparatus 6. As another example, all the series of stepsin the game processing to be performed by the game apparatus body 5 maybe performed by the terminal apparatus 6. In this case, the terminalapparatus 6 functions as a processing device that performs the steps inthe game processing, as well as a display apparatus, and therefore,there may be a game system in which: the terminal apparatus 6 is used asdisplay means while the monitor 2, the game apparatus body 5, and thecontroller 7 are not used; the board-type controller 9 is used asoperation means; and the terminal apparatus 6 is used as processingmeans. In this game system, only the terminal apparatus 6 and theboard-type controller 9 are connected wirelessly or wired, and boardoperation data is transmitted from the board-type controller 9 to theterminal apparatus 6, thereby achieving various games. Further, it isneedless to say that when the board-type controller 9 is not usedeither, the terminal apparatus 6 may be used as display means, operationmeans, and processing means.

In addition, in the above embodiment, attitude data (e.g., at least onepiece of data output from the magnetic sensor 602, the accelerationsensor 603, and the gyro sensor 604) used to calculate the attitudeand/or the motion of the terminal apparatus 6 (including the positionand the attitude per se, or changes in the position and the attitude) isoutput from the terminal apparatus 6 to the game apparatus body 5, andthe attitude and/or the motion of the terminal apparatus 6 arecalculated by the information processing performed by the game apparatusbody 5. The attitude and/or the motion of the terminal apparatus 6 to becalculated by the game apparatus body 5, however, may be calculated bythe terminal apparatus 6. In this case, the data indicating the attitudeand/or the motion of the terminal apparatus 6 that have been calculatedby the terminal apparatus 6 (i.e., data indicating the position and theattitude per se of the terminal apparatus 6, or changes in the positionand the attitude that have been calculated using the attitude data) isoutput from the terminal apparatus 6 to the game apparatus body 5, andthe data is used in the information processing performed by the gameapparatus body 5.

In addition, in the above descriptions, the terminal apparatus 6 and thegame apparatus body 5 are connected by wireless communication, and theboard-type controller 9 and the game apparatus body 5 are connected bywireless communication. Alternatively, wireless communication betweendevices may be performed in a form other than the above. As a firstexample, the terminal apparatus 6 functions as a relay device foranother wireless communication. In this case, board operation data ofthe board-type controller 9 is wirelessly transmitted to the terminalapparatus 6, and the terminal apparatus 6 wirelessly transmits, to thegame apparatus body 5, terminal operation data of the terminal apparatus6 together with the received board operation data. In this case, whilethe terminal apparatus 6 and the game apparatus body 5 are directlyconnected by wireless communication, the board-type controller 9 isconnected to the game apparatus body 5 via the terminal apparatus 6 bywireless communication. As a second example, the board-type controller 9functions as a relay device for another wireless communication. In thiscase, terminal operation data of the terminal apparatus 6 is wirelesslytransmitted to the board-type controller 9, and the board-typecontroller 9 wirelessly transmits, to the game apparatus body 5, boardoperation data of the board-type controller 9 together with the receivedterminal operation data. In this case, the board-type controller 9 andthe game apparatus body 5 are directly connected by wirelesscommunication, while the terminal apparatus 6 is connected to the gameapparatus body 5 via the board-type controller 9 by wirelesscommunication.

In addition, the terminal apparatus 6 and/or the board-type controller 9may be electrically connected to the game apparatus body 5 via cables.In this case, the cables connected to the terminal apparatus 6 and/orthe board-type controller 9 are connected to a connection terminal ofthe game apparatus body 5. As a first example, the terminal apparatus 6and the game apparatus body 5 are electrically connected via a firstcable, and the board-type controller 9 and the game apparatus body 5 areelectrically connected via a second cable. As a second example, theterminal apparatus 6 and the game apparatus body 5 are electricallyconnected via a cable. In this case, board operation data of theboard-type controller 9 may be wirelessly transmitted to the terminalapparatus 6 and then transmitted to the game apparatus body 5 via thecable. As a third example, the board-type controller 9 and the gameapparatus body 5 are electrically connected via a cable. In this case,terminal operation data of the terminal apparatus 6 may be wirelesslytransmitted to the board-type controller 9 and then transmitted to thegame apparatus body 5 via the cable. Alternatively, terminal operationdata of the terminal apparatus 6 may be wirelessly transmitted to thegame apparatus body 5 directly from the terminal apparatus 6.

In addition, in the exemplary embodiment, the game system 1 includes oneterminal apparatus 6 and one board-type controller 9. Alternatively, thegame system 1 may be configured to include a plurality of terminalapparatuses 6 and a plurality of board-type controllers 9. That is, thegame apparatus body 5 may be capable of wirelessly communicating witheach terminal apparatus 6 and each type controller 9, and may transmitgame image data, game sound data, and control data to each terminalapparatus, and receive terminal operation data, camera image data,microphone sound data, and board operation data from each terminalapparatus 6 and each board-type controller 9. When the game apparatusbody 5 wirelessly communicates with the plurality of terminalapparatuses 6 and the plurality of board-type controllers 9, the gameapparatus body 5 may perform the wireless communication in a timedivision manner or in a frequency division manner.

As described above, when the game system 1 includes a plurality ofterminal apparatuses 6 and a plurality of board-type controllers 9, aplurality of users are allowed to play more games. For example, when thegame system 1 includes two pairs of terminal apparatuses 6 andboard-type controllers 9, two users are allowed to play a gamesimultaneously. Further, when the game system 1 includes two pairs ofterminal apparatuses 6 and board-type controllers 9, the game system 1includes three display apparatuses, and therefore can generate gameimages for three users to be displayed on the respective displayapparatuses.

In addition, in the above descriptions, a plurality of load sensors 94are provided in the board-type controller 9. If, however, information ofthe position of the center of gravity of a load applied to theboard-type controller 9 is not used in the above processing, at leastone load sensor 94 may be provided in the board-type controller 9.

In addition, the exemplary embodiment is described using the stationarygame apparatus 3. The exemplary embodiment, however, may be achieved byexecuting the information processing program according to the exemplaryembodiment with an information processing apparatus such as a hand-heldgame apparatus or a general personal computer. Further, in anotherembodiment, the exemplary embodiment may be applied not only to a gameapparatus but also to a given hand-held electronic device (e.g., a PDA(Personal Digital Assistant) or a mobile telephone), a personalcomputer, a camera, and the like. Any device may be connected to theterminal apparatus 6 and the board-type controller 9 wirelessly orwired, whereby the exemplary embodiment can be achieved.

In addition, in the above descriptions, the information processing isperformed by the game apparatus body 5. At least a part of theprocessing steps in the information processing, however, may beperformed by another apparatus provided outside the game system 1. Forexample, when the game apparatus body 5 is configured to communicatewith another apparatus (e.g., a server or another game apparatus), theprocessing steps in the information processing may be performed by thegame apparatus body 5 in combination with said another apparatus. As anexample, said another apparatus performs the process of setting a playerobject, a virtual world, and the like, and data concerning the motionand the attitude of the player object is transmitted from the gameapparatus body 5 to said another apparatus, whereby the informationprocessing is performed. Then, image data indicating the virtual worldgenerated by said other apparatus is transmitted to the game apparatusbody 5, and the virtual world is displayed on the monitor 2 and the LCD61. At least a part of the processing steps in the informationprocessing is thus performed by another apparatus, whereby the sameprocessing as the information processing is achieved. It should be notedthat at least a part of the processing steps in the informationprocessing may be performed by the board-type controller 9 (themicrocomputer 100). Further, the above information processing can beperformed by one processor or by a cooperation of a plurality ofprocessors, the one processor or the plurality of processors included inan information processing system including at least one informationprocessing apparatus. Further, in the exemplary embodiment, theprocesses shown in the above flow charts are performed as a result ofthe CPU 10 of the game apparatus body 5 executing a predeterminedprogram. Alternatively, a part or all of the processes may be performedby a dedicated circuit included in the game apparatus body 5.

The systems, devices and apparatuses described herein may include one ormore processors, which may be located in one place or distributed in avariety of places communicating via one or more networks. Suchprocessor(s) can, for example, use conventional 3D graphicstransformations, virtual camera and other techniques to provideappropriate images for display. By way of example and withoutlimitation, the processors can be any of: a processor that is part of oris a separate component co-located with the stationary display and whichcommunicates remotely (e.g., wirelessly) with the movable display; or aprocessor that is part of or is a separate component co-located with themovable display and communicates remotely (e.g., wirelessly) with thestationary display or associated equipment; or a distributed processingarrangement some of which is contained within the movable displayhousing and some of which is co-located with the stationary display, thedistributed portions communicating together via a connection such as awireless or wired network; or a processor(s) located remotely (e.g., inthe cloud) from both the stationary and movable displays andcommunicating with each of them via one or more network connections; orany combination or variation of the above.

The processors can be implemented using one or more general-purposeprocessors, one or more specialized graphics processors, or combinationsof these. These may be supplemented by specifically-designed ASICs(application specific integrated circuits) and/or logic circuitry. Inthe case of a distributed processor architecture or arrangement,appropriate data exchange and transmission protocols are used to providelow latency and maintain interactivity, as will be understood by thoseskilled in the art.

Similarly, program instructions, data and other information forimplementing the systems and methods described herein may be stored inone or more on-board and/or removable memory devices. Multiple memorydevices may be part of the same device or different devices, which areco-located or remotely located with respect to each other.

In addition, the shape of the game apparatus body 5 described above, theshapes of the terminal apparatus 6, the controller 7, and the board-typecontroller 9, and the shapes, the number, the placement, or the like ofthe various operation buttons and sensors are merely illustrative, andthe exemplary embodiment can be achieved with other shapes, numbers,placements, and the like. Further, the processing orders, the settingvalues, the display forms, the criterion values, and the like that areused in the information processing described above are also merelyillustrative, and it is needless to say that the exemplary embodimentcan be achieved with other orders, display forms, and values.

In addition, the information processing program (the game program)described above may be supplied to the game apparatus body 5 not onlyfrom an external storage medium such as the optical disk 4, but also viaa wireless or wired communication link. Further, the informationprocessing program may be stored in advance in a nonvolatile storagedevice of the game apparatus body 5. It should be noted that examples ofan information storage medium for storing the information processingprogram may include a CD-ROM, a DVD, given another optical disk storagemedium similar to these, a flexible disk, a hard disk, a magneticoptical disk, and a magnetic tape, as well as a nonvolatile memory.Furthermore, the information storage medium for storing the informationprocessing program may be a nonvolatile semiconductor memory or avolatile memory. Such storage media can be defined as storage mediareadable by a computer or the like. For example, a computer or the likeis caused to read and execute programs stored in each of the storagemedia, and thereby can be caused to provide the various functionsdescribed above.

While some exemplary systems, exemplary methods, exemplary devices, andexemplary apparatuses have been described in detail, the foregoingdescription is in all aspects illustrative and not restrictive. It isunderstood that numerous other modifications and variations can bedevised without departing from the spirit and scope of the appendedclaims. It is understood that the scope of the exemplary embodimentshould be interpreted only by the scope of the appended claims. It isalso understood that one skilled in the art can implement the exemplaryembodiment in the equivalent range on the basis of the description ofthe exemplary embodiment and common technical knowledge, from thedescription of the specific embodiments. It should be understood thatwhen used in the present specification, components and the likedescribed in singular form with the words “a” and “an” before them donot exclude the plurality of these components. Furthermore, it should beunderstood that terms used in the present specification have meaningsgenerally used in the art unless otherwise specified. Therefore, unlessotherwise defined, all the jargons and technical terms have the samemeanings as those generally understood by one skilled in the art of theexemplary embodiment. In the event of any contradiction, the presentspecification (including meanings defined herein) has priority.

A storage medium having stored thereon an information processingprogram, an information processing apparatus, an information processingsystem, and an information processing method according to the exemplaryembodiment, when an image of a virtual world is displayed on a displayapparatus that allows a user to view a screen thereof while holding it,can display a suitable image on the display apparatus, regardless of thedirection in which the user is holding the display apparatus, andtherefore are suitable for use as an information processing program, aninformation processing apparatus, an information processing system, andan information processing method that perform the process of displayingan image of a virtual world.

1. A computer-readable storage medium having stored thereon aninformation processing program to be executed by a computer of aninformation processing apparatus capable of displaying an image on aportable display apparatus that outputs at least data based on anattitude and/or a motion of the portable display apparatus body, theinformation processing program causing the computer to execute:calculating, on the basis of the data output from the portable displayapparatus, a direction of rotation of the portable display apparatusabout a predetermined direction in real space; controlling an attitudeof an object, placed in a virtual world, such that the object rotates inthe direction of rotation about a direction that corresponds to thepredetermined direction and is set in the virtual world; controlling anattitude of a first virtual camera, for generating an image of thevirtual world including at least the object, such that the first virtualcamera rotates in the direction of rotation about the direction thatcorresponds to the predetermined direction and is set in the virtualworld; and displaying on the portable display apparatus a first imagerepresenting the virtual world viewed from the first virtual camera. 2.The computer-readable storage medium having stored thereon theinformation processing program according to claim 1, wherein an amountof rotation of the portable display apparatus about the predetermineddirection in real space is further calculated on the basis of the dataoutput from the portable display apparatus, the attitude of the objectis controlled such that the object rotates in the direction of rotationabout the direction that corresponds to the predetermined direction andis set in the virtual world, and by an amount of rotation based on theamount of rotation of the portable display apparatus, and the attitudeof the first virtual camera is controlled such that the first virtualcamera rotates in the direction of rotation about the direction thatcorresponds to the predetermined direction and is set in the virtualworld, and by an amount of rotation based on the amount of rotation ofthe portable display apparatus.
 3. The computer-readable storage mediumhaving stored thereon the information processing program according toclaim 2, wherein the attitude of the first virtual camera is controlledsuch that the first virtual camera rotates in the direction of rotationby the same amount of rotation as the amount of rotation of the portabledisplay apparatus.
 4. The computer-readable storage medium having storedthereon the information processing program according to claim 3, theinformation processing program further causing the computer to executecalculating, on the basis of the data output from the portable displayapparatus, a direction of gravity relative to the portable displayapparatus, wherein the direction of rotation and the amount of rotationof the portable display apparatus are calculated with respect to thedirection of gravity, and the attitude of the first virtual camera iscontrolled such that when the first image is displayed on the portabledisplay apparatus, the direction of gravity relative to the portabledisplay apparatus displaying the first image coincides with a directionof gravity in the virtual world displayed as the first image.
 5. Thecomputer-readable storage medium having stored thereon the informationprocessing program according to claim 3, wherein the object iscontrolled so as to rotate in the direction of rotation and by the sameamount of rotation as the amount of rotation of the portable displayapparatus, and to have such an attitude and position that a positionalrelationship between the object and the first virtual camera is fixed.6. The computer-readable storage medium having stored thereon theinformation processing program according to claim 3, wherein using asthe predetermined direction a predetermined axis provided in theportable display apparatus, the direction of rotation and the amount ofrotation of the portable display apparatus about the predetermined axisare calculated.
 7. The computer-readable storage medium having storedthereon the information processing program according to claim 6, whereinusing as the predetermined axis a perspective direction of a displayscreen that displays the first image on the portable display apparatus,the direction of rotation and the amount of rotation of portable displayapparatus about the perspective direction are calculated, and theattitude of the first virtual camera is controlled such that, using adirection of a line of sight of the first virtual camera as a directioncorresponding to the perspective direction, the first virtual camerarotates about the direction of the line of sight, in the direction ofrotation and by the same amount of rotation as the amount of rotation ofthe portable display apparatus.
 8. The computer-readable storage mediumhaving stored thereon the information processing program according toclaim 7, wherein further using, as the predetermined axis, each of aheight direction and a width direction of the portable display apparatusthat are orthogonal to the perspective direction and are orthogonal toeach other, the direction of rotation and the amount of rotation of theportable display apparatus about each of the height direction and thewidth direction are calculated, and the attitude of the first virtualcamera is controlled such that: the first virtual camera rotates about aheight direction orthogonal to the direction of the line of sight of thefirst virtual camera, in the same direction of rotation and by the sameamount of rotation as the direction of rotation and the amount ofrotation about the height direction of the portable display apparatus;and the first virtual camera rotates about a width direction orthogonalto the direction of the line of sight of the first virtual camera andorthogonal to the height direction orthogonal to the direction of theline of sight of the first virtual camera, in the same direction ofrotation and by the same amount of rotation as the direction of rotationand the amount of rotation of the portable display apparatus about thewidth direction of the portable display apparatus.
 9. Thecomputer-readable storage medium having stored thereon the informationprocessing program according to claim 3, wherein the attitude of thefirst virtual camera in the virtual world is controlled so as to be thesame as the attitude of the portable display apparatus in real space.10. The computer-readable storage medium having stored thereon theinformation processing program according to claim 1, wherein image dataindicating the first image is output to the portable display apparatus,and the portable display apparatus includes: an image data acquisitionunit that acquires the image data output from the information processingapparatus, and a display unit that displays the first image indicated bythe image data acquired by the image data acquisition unit.
 11. Thecomputer-readable storage medium having stored thereon the informationprocessing program according to claim 10, the information processingprogram further causing the computer to execute generating compressionimage data by compressing the image data indicating the first image,wherein the generated compression image data is output to the portabledisplay apparatus, the compression image data output from theinformation processing apparatus is acquired, the portable displayapparatus further includes a display image decompression unit thatdecompresses the compression image data to obtain the image dataindicating the first image, and the display unit displays the firstimage indicated by the image data that has been acquired by the imagedata acquisition unit and has been decompressed by the display imagedecompression unit.
 12. The computer-readable storage medium havingstored thereon the information processing program according to claim 1,wherein besides the first image, a second image representing the virtualworld viewed from a second virtual camera is further displayed onanother display apparatus connected to the information processingapparatus.
 13. The computer-readable storage medium having storedthereon the information processing program according to claim 12, theinformation processing program further causing the computer to executegenerating compression image data by compressing the image dataindicating the first image, wherein the generated compression image datais output to the portable display apparatus, and, besides thecompression image data, image data indicating the second image is outputto said another display apparatus without being compressed, and theportable display apparatus includes: an image data acquisition unit thatacquires the compression image data output from the informationprocessing apparatus; a display image decompression unit thatdecompresses the compression image data to obtain the image dataindicating the first image; and a display unit that displays the firstimage indicated by the image data that has been acquired by the imagedata acquisition unit and has been decompressed by the display imagedecompression unit.
 14. The computer-readable storage medium havingstored thereon the information processing program according to claim 12,the information processing program further causing the computer toexecute setting, on the basis of a position of the object in the virtualworld, the second virtual camera, for generating an image of the virtualworld, at a position different from a position of the first virtualcamera such that the object is included in the second image.
 15. Thecomputer-readable storage medium having stored thereon the informationprocessing program according to claim 14, wherein the second virtualcamera is set at a position further away from the object than the firstvirtual camera is from the object, and a range wider than a range of thevirtual world represented by the first image is displayed as the secondimage on said another display apparatus.
 16. The computer-readablestorage medium having stored thereon the information processing programaccording to claim 14, wherein the second virtual camera is set at aposition of viewing the object from a bird's-eye view in the virtualworld, and an image obtained by viewing from a bird's-eye view theobject placed in the virtual world is displayed as the second image onsaid another display apparatus.
 17. The computer-readable storage mediumhaving stored thereon the information processing program according toclaim 1, wherein the portable display apparatus includes at least one ofa gyro sensor and an acceleration sensor, each of which outputs databased on the attitude and/or the motion of the portable displayapparatus body, and on the basis of the data output from the at leastone of the gyro sensor and the acceleration sensor, the direction ofrotation and an amount of rotation of the portable display apparatus arecalculated.
 18. An information processing apparatus capable ofdisplaying an image on a portable display apparatus that outputs atleast data based on an attitude and/or a motion of the portable displayapparatus body, the information processing apparatus comprising: arotation information calculation unit that calculates, on the basis ofthe data output from the portable display apparatus, a direction ofrotation of the portable display apparatus about a predetermineddirection in real space; an object control unit that controls anattitude of an object, placed in a virtual world, such that the objectrotates in the direction of rotation about a direction that correspondsto the predetermined direction and is set in the virtual world; a firstvirtual camera control unit that controls an attitude of a first virtualcamera, for generating an image of the virtual world including at leastthe object, such that the first virtual camera rotates in the directionof rotation about the direction that corresponds to the predetermineddirection and is set in the virtual world; and a display control unitthat displays on the portable display apparatus a first imagerepresenting the virtual world viewed from the first virtual camera. 19.An information processing system including a plurality of apparatusesconfigured to communicate with each other, the information processingsystem capable of displaying an image on a portable display apparatusthat outputs at least data based on an attitude and/or a motion of theportable display apparatus body, the information processing systemcomprising: a rotation information calculation unit that calculates, onthe basis of the data output from the portable display apparatus, adirection of rotation of the portable display apparatus about apredetermined direction in real space; an object control unit thatcontrols an attitude of an object, placed in a virtual world, such thatthe object rotates in the direction of rotation about a direction thatcorresponds to the predetermined direction and is set in the virtualworld; a first virtual camera control unit that controls an attitude ofa first virtual camera, for generating an image of the virtual worldincluding at least the object, such that the first virtual camerarotates in the direction of rotation about the direction thatcorresponds to the predetermined direction and is set in the virtualworld; and a display control unit that displays on the portable displayapparatus a first image representing the virtual world viewed from thefirst virtual camera.
 20. An information processing method performed bya processor or a corporation of a plurality of processors included in aninformation processing system including at least one informationprocessing apparatus capable of displaying an image on a portabledisplay apparatus that outputs at least data based on an attitude and/ora motion of the portable display apparatus body, the informationprocessing method comprising: calculating, on the basis of the dataoutput from the portable display apparatus, a direction of rotation ofthe portable display apparatus about a predetermined direction in realspace; controlling an attitude of an object, placed in a virtual world,such that the object rotates in the direction of rotation about adirection that corresponds to the predetermined direction and is set inthe virtual world; controlling an attitude of a first virtual camera,for generating an image of the virtual world including at least theobject, such that the first virtual camera rotates in the direction ofrotation about the direction that corresponds to the predetermineddirection and is set in the virtual world; and displaying on theportable display apparatus a first image representing the virtual worldviewed from the first virtual camera.